ACA Secret Key Vault Reference — Logging Gap Variant (H4b) Lab¶

Reproduce the logging-gap failure surface where az containerapp secret set --identity system --key-vault-url ... fails with Unable to get value using Managed identity → Get https://login.microsoftonline.com/<tenant>/.well-known/openid-configuration: EOF, and the base H4 KQL query returns zero Deny rows — not because the firewall let the request through, but because the AzureFirewallApplicationRule diagnostic category is disabled. This lab proves that re-enabling the diagnostic category restores observability of the denial without restoring connectivity: the secret set still fails, because the Entra Application Rule is absent throughout.

This lab is a reader-generated 17-gate Phase B falsification workflow. You run trigger.sh and falsify.sh against your own Azure subscription to capture one live H0 → H1 → H2 cohort (files 01-13) into labs/aca-secret-kv-ref-mi-network-path-h4b/evidence/. You then run verify.sh, which reads only those local files (no Azure API calls) and deterministically emits the four Phase B gate JSONs (14-17) that validate the narrow claim: with the Entra Application Rule absent in both H1 and H2, the AzureFirewallApplicationRule diagnostic category enable flag is the sole controlled variable, and toggling it changes only whether the ongoing denial is visible, never whether the secret set succeeds. The lab does not ship a committed cohort; each reader generates their own.

Bounded-scope disclosure: the workflow does not prove exact stderr wording, log ingestion latency, retry cadence, component identity behind the OIDC EOF, response body shape, token caching behavior, SKU generality (Basic vs Standard/Premium firewall), or region generality. Those confounders are carried explicitly in Gate 17 explicit_drops, so readers can see exactly where the evidence ceiling stops.

Lab Metadata¶

1) Background¶

Azure Container Apps supports Key Vault references in the secret manifest: the app declares a reference of the form --key-vault-url https://<vault>.vault.azure.net/secrets/<name> and the platform resolves it using a managed identity. Before the platform can call Key Vault it must acquire a token, and managed identity token acquisition requires an OIDC discovery step against the Entra ID authority — a plain HTTPS GET https://login.microsoftonline.com/<tenant>/.well-known/openid-configuration (the client may use login.microsoft.com instead — it picks one host at runtime). In an environment with workload-profile networking and a 0.0.0.0/0 → Azure Firewall UDR on the workload subnet, that discovery request egresses through Azure Firewall. If the firewall policy carries no Application Rule permitting the Entra authority FQDN, the TLS handshake is terminated mid-stream and the caller sees an EOF — a network reset, not an HTTP status code. This is the H4 root cause, covered end-to-end in the H4a lab.

H4b layers a second, independent failure of observability on top of that connectivity failure. Azure Firewall does not log Application Rule actions unless the AzureFirewallApplicationRule diagnostic log category is enabled on a diagnostic setting that routes to the queried Log Analytics workspace. When that category is disabled:

• The firewall continues to drop the OIDC discovery packet based on policy — connectivity is broken exactly as in H4a.
• The firewall emits no log entry for that drop — observability is broken.
• Both the resource-specific AZFWApplicationRule table and the legacy AzureDiagnostics table stay empty for the failure window, even though the firewall is actively denying traffic.

This creates a logging gap. An operator who runs the base H4 diagnostic KQL sees zero Deny rows and — if they treat zero rows as proof of innocence — wrongly concludes the firewall is not the cause and chases the wrong hypothesis (Key Vault RBAC, MI configuration, DNS). The base H4a lab guide explicitly warns that "a zero-row result from the base H4 KQL does not falsify H4 by itself"; H4b is the reproducer for exactly that trap.

Two properties of firewall diagnostic logging make the reproduction precise:

• Logging is not retroactive. Re-enabling the AzureFirewallApplicationRule category in H2 does not surface the earlier H1 denial. Only denials that occur after the category is enabled are logged. Therefore H2 must generate a brand-new denial (a fresh secret-set attempt) after logging is re-enabled to capture the smoking-gun Deny row.
• Attribution must be guarded. Because the ACA control-plane worker may internally retry, a Deny row appearing shortly after re-enable could in principle belong to a late retry of the H1 attempt rather than the new H2 attempt. The lab records h2_diag_enable_iso and h2_secret_set_start_iso separately and runs a pre-H2 guard query proving zero Deny rows exist in the window between re-enable and the new attempt, so the >=1 row counted afterward is provably from the new H2 attempt.

Architecture¶

flowchart TD
    A["Operator: az containerapp secret set --identity system --key-vault-url"] --> B["ACA control-plane worker resolves KV URL"]
    B --> C["MI token acquisition begins:<br/>GET https://login.microsoftonline.com/{tenant}/.well-known/openid-configuration"]
    C --> D["Egress from workload subnet snet-aca 10.90.0.0/23"]
    D --> E{"UDR 0.0.0.0/0<br/>→ Azure Firewall private IP?"}
    E -->|Yes| F["Azure Firewall Basic<br/>Policy afwp-{baseName}-{suffix}<br/>Entra Application Rule ABSENT (H1 and H2)"]
    F --> G["OIDC discovery DENIED at the firewall<br/>TLS reset, response body EOF<br/>Secret set FAILS in both H1 and H2<br/>Revision UNCHANGED, ingress HTTP 200"]
    G --> H{"AzureFirewallApplicationRule<br/>diagnostic category enabled<br/>on diag-to-law?"}
    H -->|H0 baseline: Yes + rule present| I["Discovery succeeds, secret set exit 0<br/>kvref-h0 present"]
    H -->|H1: No| J["Deny NOT logged<br/>AZFWApplicationRule → 0 rows<br/>(logging gap)<br/>kvref-h1 absent"]
    H -->|H2: Yes, rule still absent| K["NEW denial logged<br/>AZFWApplicationRule → >=1 Deny row<br/>(observability restored, connectivity still broken)<br/>kvref-h2 absent"]

2) Hypothesis¶

IF an Azure Container Apps environment routes workload-subnet egress through an Azure Firewall via 0.0.0.0/0 UDR, and the app has a system-assigned managed identity granted Key Vault Secrets User at the target Key Vault scope, and the firewall policy's Entra Application Rule is absent, and the only variable toggled between the two experimental states is the firewall's AzureFirewallApplicationRule diagnostic category, THEN:

• H0 baseline (Entra rule present, diagnostic category enabled): az containerapp secret set --identity system --key-vault-url ... succeeds with exit code 0. The named secret kvref-h0 appears in properties.configuration.secrets with a populated keyVaultUrl field. latestReadyRevisionName is unchanged (secret set does not create a new revision).
• H1 (Entra rule removed AND diagnostic category disabled — the logging gap): The command fails with exit code non-zero. stderr carries the marker phrases Failed to update secrets and Unable to get value using Managed identity and includes the substring openid-configuration. configuration.secrets does NOT contain kvref-h1. latestReadyRevisionName is still unchanged. Ingress still returns HTTP 200 (the running data plane is not degraded; only the control-plane secret set is blocked). The firewall query over the H1 window returns 0 Deny rows for the Entra authority FQDN — the denial occurred but was never logged.
• H2 (diagnostic category re-enabled, Entra rule STILL absent — observability restored, connectivity still broken): A new secret-set attempt STILL fails with exit code non-zero. kvref-h2 is absent from configuration.secrets. latestReadyRevisionName is still unchanged from baseline. Ingress still returns HTTP 200. A pre-H2 guard query over the window between diagnostic re-enable (h2_diag_enable_iso) and the new attempt (h2_secret_set_start_iso) returns 0 rows. The firewall query over the H2 window (from h2_secret_set_start_iso) now returns ≥ 1 Deny row for the Entra authority FQDN from the workload subnet source IP.

3) Runbook¶

Prerequisites¶

• Azure CLI 2.80+ with the containerapp extension.
• Azure subscription permissions for: resource group deploy, role assignment (Microsoft.Authorization/roleAssignments/write), Container Apps management, Azure Firewall / Firewall Policy management, Key Vault management, and diagnostic-settings management (Microsoft.Insights/diagnosticSettings/write) on the firewall.
• Ability to reach the Log Analytics workspace via az monitor log-analytics query for the KQL evidence files 09, the pre-H2 guard, and 13.

Deploy infrastructure¶

export RG="rg-aca-secret-kv-ref-mi-network-path-h4b"
export LOCATION="koreacentral"
export BASE_NAME="acasech4b01"

az group create --name "$RG" --location "$LOCATION"

az deployment group create \
    --resource-group "$RG" --name aca-secret-kv-ref-mi-network-path-h4b \
    --template-file labs/aca-secret-kv-ref-mi-network-path-h4b/infra/main.bicep \
    --parameters baseName="$BASE_NAME" \
    --parameters deploymentPrincipalId="$(az ad signed-in-user show --query id --output tsv)"

Expected output:

• Resource group creation succeeds.
• Deployment provisioningState is Succeeded.
• Bicep outputs include appName, environmentName, keyVaultName, keyVaultUri, firewallPolicyName, firewallResourceId, entraAuthorityRuleCollectionName, entraAuthorityRuleName, logAnalyticsCustomerId.
• The diag-to-law diagnostic setting shows the AzureFirewallApplicationRule category enabled at baseline.

Run the H0 baseline (trigger.sh)¶

bash labs/aca-secret-kv-ref-mi-network-path-h4b/trigger.sh

Expected output:

• 04-h0-secret-set-outcome.json contains exit_code: 0.
• 05-h0-app-state-after.json shows the secret kvref-h0 present in configuration.secrets with a populated keyVaultUrl field.
• 05-h0-app-state-after.json latestReadyRevisionName matches the value in 02-h0-app-state-before.json (secret set does not create a new revision).
• 01-deployment-outputs.json includes firewall_resource_id (the --resource target used later by az monitor diagnostic-settings).

Run the H1 → H2 falsification (falsify.sh)¶

bash labs/aca-secret-kv-ref-mi-network-path-h4b/falsify.sh

Expected output:

• 06-h1-firewall-rule-removed.json confirms the rule collection is absent and records controlled_variable_observability_disabled: true.
• 07-h1-secret-set-outcome.json contains exit_code non-zero and stderr with Failed to update secrets, Unable to get value using Managed identity, and openid-configuration.
• 08-h1-app-state.json shows revision name unchanged, ingress HTTP 200, and kvref-h1 absent from configuration.secrets.
• 09-h1-firewall-deny-log-absent.json shows final_deny_row_count: 0 — the denial occurred but was not logged (the logging gap).
• 10-h2-firewall-diagnostics-enabled.json confirms the diagnostic category is re-enabled, records h2_diag_enable_iso, h2_secret_set_start_iso, pre_h2_guard_deny_row_count: 0, and controlled_variable_absent_after_h2_observability_fix: true (the Entra rule is still absent).
• 11-h2-secret-set-outcome.json contains exit_code non-zero — the secret set STILL fails.
• 12-h2-app-state.json shows revision name still unchanged from baseline, ingress HTTP 200, and kvref-h2 absent from configuration.secrets.
• 13-h2-firewall-deny-log.json shows final_deny_row_count >= 1 — the new denial is now visible.

Run the offline verifier over your locally generated pack¶

bash labs/aca-secret-kv-ref-mi-network-path-h4b/verify.sh

Expected output:

• 17/17 gate passes on a valid cohort.
• evidence/14-cohort-integrity-gate.json shows g_baseline_revision_silence_invariant_holds: true (proves latestReadyRevisionName is identical across snapshots 02, 05, 08, 12) and e_h1_start_precedes_h2_diag_enable_and_h2_attempt: true (proves the H1 window precedes the H2 re-enable, which precedes the new H2 attempt).
• evidence/15-...-gate.json shows the H1 logging-gap sub-gates pass, including h1_deny_count == 0.
• evidence/16-h2-observability-restored-gate.json shows the H2 sub-gates pass: diagnostic category enabled, Entra rule absent, secret set still fails, kvref-h2 absent, ingress 200, revision unchanged, and h2_deny_count >= 1.
• evidence/17-bounded-falsification-gate.json enumerates the eight documented explicit drops (stderr wording, log ingestion latency, retry cadence, component identity, response body shape, token caching, SKU generality, region generality) and classifies the H1↔H2 flip as an observability restoration, not a connectivity fix.

Optional: toggle the diagnostic category manually¶

The falsify.sh script already toggles the category. If you want to re-enable the AzureFirewallApplicationRule category independently after H1 disabled it:

FIREWALL_RESOURCE_ID="$(jq -r .firewall_resource_id labs/aca-secret-kv-ref-mi-network-path-h4b/evidence/01-deployment-outputs.json)"

az monitor diagnostic-settings update \
    --name diag-to-law \
    --resource "$FIREWALL_RESOURCE_ID" \
    --logs '[{"category":"AzureFirewallApplicationRule","enabled":true},{"category":"AzureFirewallNetworkRule","enabled":true}]'

Expected output:

• The command returns the updated diagnostic-settings object with AzureFirewallApplicationRule enabled: true.
• A new az containerapp secret set attempt still fails (the Entra rule remains absent), but the resulting denial now appears in AZFWApplicationRule within ~60 seconds of ingestion.

4) Experiment Log¶

Why 0 rows is not proof of innocence¶

The central lesson of this lab is that the H1 KQL result — zero Deny rows — is produced by a firewall that is actively dropping the OIDC discovery request. The zero-row result is a property of the diagnostic pipeline (category disabled), not of the firewall's policy decision (deny). An operator who stops at "0 rows in AZFWApplicationRule, therefore not a firewall problem" has been fooled by the logging gap. H2 proves this directly: the same policy state (Entra rule absent) that produced zero rows in H1 produces a visible Deny row in H2 once the diagnostic category is re-enabled — nothing about connectivity changed, only whether the denial was recorded. This is why the Playbook H4 Variants preflight requires confirming the diagnostic category is enabled before treating a zero-row result as disproof of H4.

Silence-gate invariant — why it is central to this lab¶

Unlike most Container Apps failure labs where the failure mode creates a new revision, a secret set operation never creates a new revision. Both configuration.secrets and the running data plane are managed under the same latestReadyRevisionName throughout H0, H1, and H2. This means:

1. The latestReadyRevisionName observed in 02-h0-app-state-before.json must equal the value in 05-h0-app-state-after.json, 08-h1-app-state.json, AND 12-h2-app-state.json. Gate 14 sub-gate g_baseline_revision_silence_invariant_holds proves this.
2. The ingress FQDN must return HTTP 200 throughout — the data plane is not degraded by a secret-set failure in either H1 or H2.
3. The failure signal is only visible in the exit code and stderr of az containerapp secret set, and — once the diagnostic category is enabled — in the firewall Deny log. In H1 the failure is doubly quiet: no revision change, no ingress downtime, AND no firewall log. That double silence is exactly what makes the logging gap dangerous.

Operators debugging real customer reports of this failure often start by looking at the running app and see nothing wrong, then look at the firewall logs and see nothing either, and prematurely conclude the report is spurious. The silence-gate invariant plus the logging-gap trap together prove that neither a healthy running data plane nor an empty firewall log is evidence against this failure.

5) Verification Queries¶

KQL: firewall log for Entra authority FQDN (H1, pre-H2 guard, and H2 windows)¶

Run this query three times with different time bounds: the H1 window (expect 0 rows), the pre-H2 guard window from h2_diag_enable_iso to h2_secret_set_start_iso (expect 0 rows), and the H2 window from h2_secret_set_start_iso onward (expect ≥ 1 Deny row).

let AcaSubnet = "10.90.0.0/23";
let EntraFqdns = dynamic(["login.microsoftonline.com", "login.microsoft.com"]);
let StartTime = todatetime("<window start ISO8601>");
let EndTime   = todatetime("<window end ISO8601>");
union isfuzzy=true
    (AZFWApplicationRule
        | where TimeGenerated between (StartTime .. EndTime)
        | where Fqdn has_any (EntraFqdns)
        | where ipv4_is_in_range(SourceIp, AcaSubnet)
        | project TimeGenerated, Action, Fqdn, SourceIp, TargetUrl, Policy, RuleCollection, Rule),
    (AzureDiagnostics
        | where TimeGenerated between (StartTime .. EndTime)
        | where Category == "AzureFirewallApplicationRule"
        | where msg_s has_any (EntraFqdns)
        | extend Action = extract(@"Action: (\w+)", 1, msg_s)
        | extend SourceIp = extract(@"from (\d+\.\d+\.\d+\.\d+):", 1, msg_s)
        | extend Fqdn = extract(@"Url: https?://([^/]+)", 1, msg_s)
        | where ipv4_is_in_range(SourceIp, AcaSubnet)
        | project TimeGenerated, Action, Fqdn, SourceIp, msg_s)
| order by TimeGenerated asc

Expected interpretation:

• H1 window: [Observed] 0 rows. The firewall dropped the OIDC discovery request, but the AzureFirewallApplicationRule category was disabled, so nothing was logged. This zero-row result is the logging gap — not evidence the firewall allowed the request.
• Pre-H2 guard window (h2_diag_enable_iso → h2_secret_set_start_iso): [Observed] 0 rows. This proves firewall logging is not retroactive (the earlier H1 denial did not appear when logging was re-enabled) and guards against misattributing a late-flushed H1 retry to the H2 attempt.
• H2 window (from h2_secret_set_start_iso): [Measured] ≥ 1 row with Action == "Deny" and Fqdn in the two-host Entra authority set from a source IP inside 10.90.0.0/23. [Inferred] Because the Entra rule was never restored, this Deny row confirms the denial was ongoing the whole time and only became visible once the diagnostic category was re-enabled.

Both login.microsoftonline.com and login.microsoft.com are documented Entra authority endpoints; the OIDC discovery client may use either host, so has_any over the EntraFqdns list keeps the query robust. The union isfuzzy=true over AZFWApplicationRule (resource-specific schema) and AzureDiagnostics (legacy schema with msg_s free text) tolerates either firewall diagnostic delivery mode. [Not Proven] This query does not prove the exact ingestion latency; it only proves that, after a ≥ 60s wait, the Deny row is present in H2 and absent in H1.

KQL: confirm the diagnostic category state at query time¶

Before treating an empty result as disproof, confirm whether the AzureFirewallApplicationRule category is actually enabled on the firewall's diagnostic setting. This is a control-plane check, not a log query:

FIREWALL_RESOURCE_ID="$(jq -r .firewall_resource_id labs/aca-secret-kv-ref-mi-network-path-h4b/evidence/01-deployment-outputs.json)"

az monitor diagnostic-settings list \
    --resource "$FIREWALL_RESOURCE_ID" \
    --query "value[].logs[?category=='AzureFirewallApplicationRule'].{category:category, enabled:enabled}"

Expected:

• [Observed] During H1: the category's enabled flag is false — the query result explains the zero-row KQL result.
• [Observed] During H0 and H2: the flag is true.

6) Portal Evidence (reader-generated captures)¶

Azure Portal screenshots to collect after running the H0 → H1 → H2 sequence. Save to docs/assets/troubleshooting/aca-secret-kv-ref-mi-network-path-h4b/ with the exact filenames listed in the Screenshot capture checklist below. Follow the PII-replacement rules in the repository AGENTS.md — use text replacement (not black-box redaction) for subscription IDs, tenant names, and user emails.

Container App — Secrets blade (H0 baseline)¶

Container App — Revisions blade (silence gate)¶

Firewall — Diagnostic settings (H1 category disabled)¶

Log Analytics — zero Deny rows during H1 (the logging gap)¶

Firewall — Diagnostic settings (H2 category re-enabled)¶

Log Analytics — Firewall Deny row during H2 (observability restored)¶

Screenshot capture checklist¶

Clean Up¶

bash labs/aca-secret-kv-ref-mi-network-path-h4b/cleanup.sh

Related Playbook¶

• Secret and Key Vault Reference Failure — H4 Variants: When Base H4 KQL Returns Zero Rows (H4b row)

See Also¶

• ACA Secret Key Vault Reference — Managed Identity Network Path Lab (H4a base variant) (proves the connectivity fix this variant does not apply)
• Managed Identity Auth Failure Playbook
• Managed Identity Key Vault Failure Lab (sibling lab — proves the RBAC-missing path)
• Egress Control

Sources¶

• Azure Firewall logs and metrics
• Manage secrets in Azure Container Apps
• Managed identities in Azure Container Apps
• Configure UDR with Azure Firewall to lock down outbound traffic
• Securing a custom VNET in Azure Container Apps with NSG and Azure Firewall