ACA Secret Key Vault Reference — Azure Firewall Premium TLS-Inspection Variant (H4g) Lab¶

Cost warning This lab uses Azure Firewall Premium, which is billed at a significantly higher rate than the other H4 variants and is expected to cost several US dollars even for a short 1-2 hour run. Delete the resource group immediately after capturing evidence.

Reproduce the Azure Firewall Premium TLS-inspection failure surface where az containerapp secret set --secrets <name>=keyvaultref:<url>,identityref:system fails only while the Entra-authority application rule for login.microsoftonline.com and login.microsoft.com has terminateTLS=true. This is the scripted equivalent of using a system-assigned managed identity plus a Key Vault reference. The topology contains Azure Firewall Premium, a Firewall Policy Premium with TLS inspection configured through an intermediate CA certificate, and a route table that sends 0.0.0.0/0 from the ACA workload subnet to the firewall private IP. The only controlled variable is the Entra-authority rule's TLS-inspection flag.

This lab is a reader-generated 17-gate Phase B falsification workflow. You run trigger.sh and falsify.sh against your own Azure subscription to capture one live H0 → H1 → H2 cohort (files 01-13) into labs/aca-secret-kv-ref-mi-network-path-h4g/evidence/. You then run verify.sh, which reads only those local files (no Azure API calls) and deterministically emits the four Phase B gate JSONs (14-17) that validate the narrow claim: TLS inspection of the Entra authority FQDNs is the sole controlled variable while app, Key Vault, identity, RBAC, firewall presence, route table, revision, ingress, NSG, and DNS stay constant.

Bounded-scope disclosure: this workflow does not prove a control-plane packet capture, does not directly observe the control-plane TLS chain, does not prove workload and control-plane egress are identical, and does not generalize beyond the two exercised Entra authority FQDNs. It also makes no third-party-NVA claim and no Key Vault data-plane-failure claim. Those confounders are carried explicitly in Gate 17 explicit_drops.

Lab Metadata¶

1) Background¶

Azure Container Apps supports Key Vault references in the secret manifest: the app declares a reference of the form --key-vault-url https://<vault>.vault.azure.net/secrets/<name> and the platform resolves it using a managed identity. Before the platform can call Key Vault it must acquire a token, and managed identity token acquisition requires an OIDC discovery step against the Entra authority — a plain HTTPS request to https://login.microsoftonline.com/<tenant>/.well-known/openid-configuration (the client may use login.microsoft.com instead — it picks one host at runtime).

H4g reproduces a TLS-inspection exemption failure. Azure Firewall Premium and the route table both remain present throughout the lab. Key Vault stays public, RBAC stays constant, the app stays on the same revision, ingress keeps serving HTTP 200, and no NSG or DNS variable changes. What changes is whether the Entra-authority application rule terminates TLS:

• [Observed] H0 succeeds when the Entra-authority rule exists with terminateTLS=false.
• [Observed] H1 fails when that same rule exists with terminateTLS=true.
• [Observed] H2 succeeds again when that same rule is restored to terminateTLS=false.
• [Observed] A workload replica can capture the interception CA chain during H1 and its absence during H2 with openssl s_client.
• [Strongly Suggested] The control-plane secret resolver is affected by the same Entra-authority TLS-inspection exemption because the secret-set result flips exactly with the same H0/H1/H2 variable.
• [Not Proven] The lab never directly captures the control-plane TLS chain.

Architecture¶

flowchart TD
    A["Operator: az containerapp secret set --secrets <name>=keyvaultref:<url>,identityref:system"] --> B["ACA control-plane worker resolves Key Vault reference"]
    B --> C["Managed identity OIDC discovery must reach login.microsoftonline.com / login.microsoft.com"]
    C --> D["UDR on snet-aca: 0.0.0.0/0 -> Azure Firewall private IP"]
    D --> E["Azure Firewall Premium + Firewall Policy Premium"]
    E --> F{"Entra-authority rule terminateTLS?"}
    F -->|H0 / H2: false| G["TLS-inspection exemption for the Entra authority\nsecret set succeeds\nrevision unchanged"]
    F -->|H1: true| H["Firewall terminates TLS for the Entra authority\nsecret set fails with MI / OIDC + TLS or cert clue\nrevision unchanged, ingress HTTP 200"]
    H --> I["Workload openssl capture can show the lab interception CA chain"]

2) Hypothesis¶

IF an Azure Container Apps environment uses Azure-provided DNS on its infrastructure VNet, the app has a system-assigned managed identity granted Key Vault Secrets User at the target Key Vault scope, Azure Firewall Premium is present with Firewall Policy Premium TLS inspection configured, the ACA subnet has a route table sending 0.0.0.0/0 to the firewall private IP, and the only changed field is the Entra-authority rule's terminateTLS flag, THEN:

• H0 baseline (terminateTLS=false): az containerapp secret set --secrets kvref-h0=keyvaultref:<url>,identityref:system succeeds with exit code 0. The named secret kvref-h0 appears in properties.configuration.secrets. latestReadyRevisionName is unchanged.
• H1 trigger (terminateTLS=true): after redeploying the same Firewall Policy so the same Entra-authority rule terminates TLS, the command fails with exit code non-zero. stderr carries a classifier-friendly signature: (managed-identity clue OR OIDC clue) AND (TLS/certificate clue). configuration.secrets does not contain kvref-h1. latestReadyRevisionName is still unchanged. Ingress still returns HTTP 200. [Observed] the workload openssl capture shows the unexpected lab interception CA chain.
• H2 fix (terminateTLS=false): after redeploying the same Firewall Policy so the same Entra-authority rule again exempts TLS inspection, a new secret-set attempt succeeds with exit code 0. kvref-h2 appears in configuration.secrets. latestReadyRevisionName is still unchanged from baseline. Ingress still returns HTTP 200. [Observed] the workload openssl capture no longer shows the lab interception CA chain.

3) Runbook¶

Prerequisites¶

• Azure CLI 2.80+ with the containerapp extension.
• Azure subscription permissions for: resource group deploy, role assignment (Microsoft.Authorization/roleAssignments/write), Container Apps management, Key Vault management, Azure Firewall / Firewall Policy management, route-table management, and user-assigned managed identity creation.
• jq, curl, and a local shell that can run the lab scripts.
• A password-less PFX intermediate CA certificate stored as a Key Vault secret for Azure Firewall Premium TLS inspection.
• A pre-created user-assigned managed identity that already has Get and List access to the Key Vault that stores that CA secret.

Pre-create the TLS-inspection identity access path¶

Before you deploy main.bicep, complete these reader-owned steps:

1. Create a user-assigned managed identity in Azure.
2. Grant that identity Get and List access on the Key Vault that stores the TLS-inspection intermediate CA secret.
3. Export that identity resource ID as TLS_INSPECTION_IDENTITY_RESOURCE_ID and pass it into the lab deployment.

Deploy infrastructure¶

export RG="rg-aca-secret-kv-ref-mi-network-path-h4g"
export LOCATION="koreacentral"
export BASE_NAME="acasech4g01"
export TLS_INSPECTION_CA_KEY_VAULT_SECRET_ID="https://<ca-vault>.vault.azure.net/secrets/<ca-secret>/<version>"
export TLS_INSPECTION_CA_CERTIFICATE_NAME="lab-h4g-intermediate-ca"
export TLS_INSPECTION_IDENTITY_RESOURCE_ID="/subscriptions/<subscription-id>/resourceGroups/<identity-rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<identity-name>"

az group create --name "$RG" --location "$LOCATION"

az deployment group create \
    --resource-group "$RG" \
    --name aca-secret-kv-ref-mi-network-path-h4g \
    --template-file labs/aca-secret-kv-ref-mi-network-path-h4g/infra/main.bicep \
    --parameters baseName="$BASE_NAME" \
    --parameters deploymentPrincipalId="$(az ad signed-in-user show --query id --output tsv)" \
    --parameters tlsInspectionCaKeyVaultSecretId="$TLS_INSPECTION_CA_KEY_VAULT_SECRET_ID" \
    --parameters tlsInspectionCaCertificateName="$TLS_INSPECTION_CA_CERTIFICATE_NAME" \
    --parameters tlsInspectionIdentityResourceId="$TLS_INSPECTION_IDENTITY_RESOURCE_ID"

Expected output:

• Resource group creation succeeds.
• Deployment provisioningState is Succeeded.
• Bicep outputs include azureFirewallPresent=true, azureFirewallSku="Premium", firewallPolicyPresent=true, tlsInspectionConfigured=true, routeTableAttached=true, and the Entra-authority rule identifiers.

Run the H0 baseline (trigger.sh)¶

bash labs/aca-secret-kv-ref-mi-network-path-h4g/trigger.sh

Expected output:

• 04-h0-secret-set-outcome.json contains exit_code: 0.
• 05-h0-app-state-after.json shows the secret kvref-h0 present in configuration.secrets.
• 01-deployment-outputs.json shows azure_firewall_present: true, azure_firewall_sku: "Premium", firewall_policy_present: true, tls_inspection_configured: true, route_table_attached: true, entra_authority_terminate_tls: false, nsg_deny_present: false, dns_override_present: false, and vwan_routing_intent_present: false.

Run the H1 → H2 falsification (falsify.sh)¶

bash labs/aca-secret-kv-ref-mi-network-path-h4g/falsify.sh

Expected output:

• 06-h1-entra-rule-updated.json confirms the Entra-authority rule exists with terminateTLS: true.
• 07-h1-secret-set-outcome.json contains exit_code non-zero and classifier-friendly stderr evidence.
• 08-h1-app-state.json shows revision unchanged, ingress HTTP 200, and kvref-h1 absent.
• 09-h1-rule-state-and-openssl.json includes the H1 firewall-rule snapshot plus a reader-generated workload openssl placeholder that must be completed with status: "completed" and reader_asserted_contains_lab_intermediate_ca: true.
• 10-h2-entra-rule-updated.json confirms the Entra-authority rule exists with terminateTLS: false.
• 11-h2-secret-set-outcome.json contains exit_code: 0.
• 12-h2-app-state.json shows revision unchanged, ingress HTTP 200, and kvref-h2 present.
• 13-h2-rule-state-and-openssl.json includes the H2 firewall-rule snapshot plus a reader-generated workload openssl placeholder that must be completed with status: "completed" and reader_asserted_contains_lab_intermediate_ca: false.

Run the offline verifier over your locally generated pack¶

bash labs/aca-secret-kv-ref-mi-network-path-h4g/verify.sh

Expected output:

• 17/17 gate passes on a valid cohort.
• evidence/14-cohort-integrity-gate.json shows the Firewall Premium / TLS-inspection / route-table anchors and the revision silence invariant.
• evidence/15-h1-tls-inspection-produces-failure-gate.json proves H1 terminateTLS=true, the classifier match, and the workload-CA observation.
• evidence/16-h2-exemption-restores-success-gate.json proves H2 terminateTLS=false, recovery, and the workload-CA disappearance.
• evidence/17-bounded-falsification-gate.json concludes only that TLS inspection of the Entra authority FQDNs is necessary and sufficient to reproduce this lab's secret-resolution failure.

Optional: inspect the Firewall Policy rule state manually¶

az network firewall policy rule-collection-group show \
    --resource-group "$RG" \
    --policy-name "$(jq -r .firewall_policy_name labs/aca-secret-kv-ref-mi-network-path-h4g/evidence/01-deployment-outputs.json)" \
    --name "$(jq -r .firewall_rule_collection_group_name labs/aca-secret-kv-ref-mi-network-path-h4g/evidence/01-deployment-outputs.json)"

Expected interpretation:

• H1 window: [Observed] the Entra-authority application rule exists and has terminateTLS: true.
• H2 window: [Observed] the same rule exists and has terminateTLS: false.
• [Not Proven] This query alone does not prove the control-plane TLS chain. It proves the configured Firewall Policy state.

4) Experiment Log¶

5) Verification Queries¶

CLI: inspect the Entra-authority Firewall Policy rule¶

az network firewall policy rule-collection-group show \
    --resource-group "$RG" \
    --policy-name "$(jq -r .firewall_policy_name labs/aca-secret-kv-ref-mi-network-path-h4g/evidence/01-deployment-outputs.json)" \
    --name "$(jq -r .firewall_rule_collection_group_name labs/aca-secret-kv-ref-mi-network-path-h4g/evidence/01-deployment-outputs.json)"

Expected interpretation:

• H1 window: [Observed] terminateTLS is true for the Entra-authority rule.
• H2 window: [Observed] terminateTLS is false for the same rule.
• [Not Proven] This query is stronger than a prose summary but still proves only the configured Firewall Policy state, not the control-plane TLS chain.

CLI: reader-run workload openssl capture¶

az containerapp exec \
    --name "$(jq -r .app_name labs/aca-secret-kv-ref-mi-network-path-h4g/evidence/01-deployment-outputs.json)" \
    --resource-group "$RG" \
    --command "sh -lc 'openssl s_client -connect login.microsoftonline.com:443 -servername login.microsoftonline.com -showcerts < /dev/null 2>&1'"

Expected interpretation:

• H1 window: [Observed] the workload capture shows the lab interception CA chain.
• H2 window: [Observed] the workload capture no longer shows that interception CA.
• [Strongly Suggested] If the workload chain flips together with the secret-set result, the control-plane secret resolver is strongly suggested to be affected by the same Entra-authority exemption.
• [Not Proven] This remains workload evidence, not direct control-plane TLS-chain proof.

6) Portal Evidence (reader-generated captures)¶

Azure Portal screenshots to collect after running the H0 → H1 → H2 sequence. Save to docs/assets/troubleshooting/aca-secret-kv-ref-mi-network-path-h4g/ with the exact filenames listed in the checklist below.

Recommended capture set¶

Portal-evidence claim ceiling reminders:

• [Observed] Portal screenshots can prove the configured Firewall Policy rule state, revision continuity, and secret presence/absence.
• [Strongly Suggested] The secret-resolver path is strongly suggested to be affected by the same Entra-authority TLS-inspection exemption because H0/H1/H2 behavior flips with only the terminateTLS field.
• [Not Proven] Portal screenshots do not directly prove the control-plane TLS chain.

Clean Up¶

bash labs/aca-secret-kv-ref-mi-network-path-h4g/cleanup.sh

Related Playbook¶

• Secret and Key Vault Reference Failure — H4 Variants: When Base H4 KQL Returns Zero Rows (future H4g row)

See Also¶

• ACA Secret Key Vault Reference — Managed Identity Network Path Lab (H4a base variant)
• ACA Secret Key Vault Reference — NSG Deny Variant (H4c)
• ACA Secret Key Vault Reference — Virtual WAN + Routing Intent Variant (H4d)
• Managed Identity Auth Failure Playbook
• Egress Control

Sources¶

• Manage secrets in Azure Container Apps
• Managed identities in Azure Container Apps
• Networking in Azure Container Apps environment
• Azure Firewall Premium certificates
• Microsoft.Network/firewallPolicies template reference