Secret and Key Vault Reference Failure¶

1. Summary¶

Symptom¶

Revisions fail or apps crash because secrets are missing, stale, or inaccessible through Key Vault references. Common symptoms include secret resolution errors during revision startup, missing configuration values in application logs, authentication failures, and secret updates that do not appear in running revision behavior.

Why this scenario is confusing¶

Secret failures span configuration mapping, revision lifecycle behavior, and Key Vault authorization. A successful secret set operation does not mean the running revision is already using the new value, and a valid managed identity does not mean that Key Vault access rights are correct.

Troubleshooting decision flow¶

flowchart TD
    A[Secret-related failure observed] --> B{secretRef exists in app secret set?}
    B -->|No| C[Create or fix secret name mapping]
    B -->|Yes| D{Key Vault reference used?}
    D -->|Yes| E[Validate identity and Key Vault permissions]
    D -->|No| F[Check revision freshness after secret update]
    C --> G[Deploy new revision and test]
    E --> G
    F --> G

2. Common Misreadings¶

• "Secret set command succeeded, so app uses it immediately." Secret updates require a new revision or restart path.
• "Key Vault reference means no RBAC checks." Managed identity still needs Key Vault access rights.

3. Competing Hypotheses¶

4. What to Check First¶

Metrics¶

• Deployment failure count and configuration error trend.

Logs¶

let AppName = "ca-myapp";
ContainerAppSystemLogs_CL
| where ContainerAppName_s == AppName
| where Log_s has_any ("secret", "KeyVault", "vault", "reference", "denied")
| project TimeGenerated, RevisionName_s, Reason_s, Log_s
| order by TimeGenerated desc

Platform Signals¶

az containerapp secret list --name "$APP_NAME" --resource-group "$RG"
az containerapp show --name "$APP_NAME" --resource-group "$RG" --query "properties.template.containers[0].env" --output json
az containerapp show --name "$APP_NAME" --resource-group "$RG" --query "identity" --output json

5. Evidence to Collect¶

Required Evidence¶

Useful Context¶

• Whether the app uses direct secrets or Key Vault references.
• Whether the secret was recently rotated or renamed.
• Whether the app is using the expected revision after the secret change.

Observed provisioning baseline:

$ az containerapp show --name "$APP_NAME" --resource-group "$RG" --query provisioningState
"Succeeded"

6. Validation and Disproof by Hypothesis¶

H1: Missing secret or wrong secretRef¶

Signals that support:

• Provisioning logs mention unresolved secret.
• secretRef names in environment configuration do not match the app secret set.
• az containerapp secret list --name "$APP_NAME" --resource-group "$RG" does not contain the exact referenced name.

Signals that weaken:

• secret list contains the exact referenced name.
• Environment mapping shows the expected secretRef values.

What to verify:

az containerapp secret list --name "$APP_NAME" --resource-group "$RG"
az containerapp show --name "$APP_NAME" --resource-group "$RG" --query "properties.template.containers[0].env" --output json

Disproof logic:

If every secretRef exactly matches a defined secret and system logs do not report unresolved secret errors, this hypothesis is weakened.

H2: Key Vault access denied¶

Signals that support:

• 403 from Key Vault and identity token success.
• Managed identity exists, but vault access does not work.
• System logs include KeyVault, vault, reference, or denied messages.

Signals that weaken:

• Vault access works with the same identity.
• Role assignment output confirms expected access scope.
• The referenced secret is enabled and accessible.

What to verify:

az containerapp show --name "$APP_NAME" --resource-group "$RG" --query "identity" --output json
az keyvault secret show --vault-name "my-kv" --name "my-secret" --query "attributes.enabled" --output tsv
az role assignment list --scope "$(az keyvault show --name "my-kv" --resource-group "$RG" --query id --output tsv)" --assignee "$(az containerapp show --name "$APP_NAME" --resource-group "$RG" --query identity.principalId --output tsv)" --output table

let AppName = "ca-myapp";
ContainerAppSystemLogs_CL
| where ContainerAppName_s == AppName
| where Log_s has_any ("secret", "KeyVault", "vault", "reference", "denied")
| project TimeGenerated, RevisionName_s, Reason_s, Log_s
| order by TimeGenerated desc

Disproof logic:

If the same identity can access the vault, the secret is enabled, and role assignment output matches expectations without 403 evidence, access denial becomes less likely.

H3: Stale revision after secret change¶

Signals that support:

• Behavior remains unchanged until a new revision deploys.
• Secret updates do not appear in running revision behavior.
• Revision list shows that the expected new revision is not active.

Signals that weaken:

• New revision already active with updated value.
• Current behavior matches the new secret value.

What to verify:

az containerapp revision list --name "$APP_NAME" --resource-group "$RG" --output table

Disproof logic:

If a new revision is already active and behavior reflects the updated value, stale revision state is not the primary cause.

H4: Managed Identity OIDC Discovery Blocked by Egress Control (UDR → Azure Firewall)¶

Signals that support:

• az containerapp secret set --identity system --key-vault-url ... (or --identity <user-assigned-id>) fails with exit code non-zero.
• stderr contains the marker phrases Failed to update secrets and Unable to get value using Managed identity.
• stderr includes the substring .well-known/openid-configuration: on either login.microsoftonline.com or login.microsoft.com (the OIDC discovery client picks one host at runtime), followed by a transport-level error (EOF, connection reset, TLS handshake failure, or timeout) rather than an HTTP status code from Microsoft Entra ID.
• The Container Apps environment uses workload-profile networking with a 0.0.0.0/0 → Azure Firewall private IP UDR on the workload subnet.
• Azure Firewall policy does not carry an Application Rule that permits login.microsoftonline.com and login.microsoft.com from the workload subnet source range.
• Azure Firewall application-rule log carries a Deny action row for either login.microsoftonline.com or login.microsoft.com (whichever host the OIDC discovery client happened to try) from a source IP inside the workload subnet, timestamped inside the failure window.
• The managed identity is correctly assigned to the Container App (principalId present) and holds Key Vault Secrets User at the target Key Vault scope; the Key Vault firewall / private endpoint allows the caller's outbound path.
• The failure is only visible in the control-plane secret-set output. latestReadyRevisionName does not change, ingress continues to return HTTP 200, and the app remains Healthy/Running (silence gate).

Signals that weaken:

• Direct probes from the same egress path to both https://login.microsoftonline.com/<tenant>/.well-known/openid-configuration and https://login.microsoft.com/<tenant>/.well-known/openid-configuration succeed and return discovery JSON. Probing only one host is not sufficient to weaken H4 because the OIDC discovery client may pick either host at runtime — a firewall rule that permits only one Entra authority FQDN will silently drop discovery calls that pick the other, so the weakening evidence must confirm both hosts are reachable.
• No firewall or NVA is on the workload subnet egress path (for example, a Consumption-only environment without UDR, or a workload-profile environment with default internet-bound egress).
• The firewall policy already contains an Application Rule that permits the Entra authority FQDNs from the workload subnet, and the firewall log shows an Allow row for the same destination in the failure window.
• The failure signature is an HTTP 401 or 403 from Microsoft Entra ID or from Key Vault rather than a transport-level EOF on the OIDC discovery URL. HTTP status codes point to H2 (Key Vault access denied) or a token audience / SDK config issue, not to a network path drop.

What to verify:

# 1. Confirm the failure surface (marker strings in stderr)
az containerapp secret set --name "$APP_NAME" --resource-group "$RG" \
    --secrets "kvref-diag=keyvaultref:https://<vault>.vault.azure.net/secrets/<name>,identityref:system" \
    2> /tmp/secret-set-stderr.txt || true
grep -E "Failed to update secrets|Unable to get value using Managed identity|openid-configuration" /tmp/secret-set-stderr.txt

# 2. Confirm the workload subnet routes 0.0.0.0/0 through Azure Firewall
INFRA_SUBNET_ID="$(az containerapp env show --name "$ACA_ENV_NAME" --resource-group "$RG" \
    --query "properties.vnetConfiguration.infrastructureSubnetId" --output tsv)"
ROUTE_TABLE_ID="$(az network vnet subnet show --ids "$INFRA_SUBNET_ID" \
    --query "routeTable.id" --output tsv)"
if [ -n "$ROUTE_TABLE_ID" ]; then
    az network route-table route list --route-table-name "$(basename "$ROUTE_TABLE_ID")" \
        --resource-group "$RG" --output table
fi

# 3. Confirm the firewall policy Application Rules for the Entra authority FQDNs
az network firewall policy rule-collection-group collection list \
    --policy-name "$FIREWALL_POLICY_NAME" \
    --resource-group "$RG" \
    --rule-collection-group-name "$FIREWALL_POLICY_RCG_NAME" \
    --query "[?ruleCollectionType=='FirewallPolicyFilterRuleCollection'].{name:name, rules:rules[?ruleType=='ApplicationRule'].{name:name, targetFqdns:targetFqdns}}" \
    --output json

# 4. Confirm the MI, RBAC, and Key Vault network path are all healthy
#    (rules out the non-H4 hypotheses: missing identity, missing role, or
#     Key Vault firewall blocking the caller). If any of these three
#     surfaces are broken, the failure is not H4.
APP_MI_PRINCIPAL_ID="$(az containerapp show --name "$APP_NAME" --resource-group "$RG" \
    --query "identity.principalId" --output tsv)"
echo "Managed identity principalId: $APP_MI_PRINCIPAL_ID"
KEY_VAULT_ID="$(az keyvault show --name "$KEY_VAULT_NAME" --resource-group "$RG" \
    --query "id" --output tsv)"
az role assignment list --assignee "$APP_MI_PRINCIPAL_ID" --scope "$KEY_VAULT_ID" \
    --query "[].{roleDefinitionName:roleDefinitionName, scope:scope}" --output table
az keyvault show --name "$KEY_VAULT_NAME" --resource-group "$RG" \
    --query "{networkAcls:properties.networkAcls, publicNetworkAccess:properties.publicNetworkAccess}" \
    --output json

// Firewall log for Entra authority FQDN during the failure window (schema-tolerant, host-tolerant)
let AcaSubnet = "10.90.0.0/23"; // workload subnet CIDR
let EntraFqdns = dynamic(["login.microsoftonline.com", "login.microsoft.com"]);
let StartTime = todatetime("<failure window start ISO8601>");
let EndTime   = todatetime("<failure window end ISO8601>");
union isfuzzy=true
    (AZFWApplicationRule
        | where TimeGenerated between (StartTime .. EndTime)
        | where Fqdn has_any (EntraFqdns)
        | where ipv4_is_in_range(SourceIp, AcaSubnet)
        | project TimeGenerated, Action, Fqdn, SourceIp, Policy, RuleCollection, Rule),
    (AzureDiagnostics
        | where TimeGenerated between (StartTime .. EndTime)
        | where Category == "AzureFirewallApplicationRule"
        | where msg_s has_any (EntraFqdns)
        | extend Action = extract(@"Action: (\w+)", 1, msg_s)
        | extend SourceIp = extract(@"from (\d+\.\d+\.\d+\.\d+):", 1, msg_s)
        | extend Fqdn = extract(@"Url: https?://([^/]+)", 1, msg_s)
        | where ipv4_is_in_range(SourceIp, AcaSubnet)
        | project TimeGenerated, Action, Fqdn, SourceIp, msg_s)
| order by TimeGenerated asc

Both login.microsoftonline.com and login.microsoft.com are documented Entra authority endpoints; the OIDC discovery client may use either host, and matching only one silently misses drops on the other.

Disproof logic:

The following observations weaken H4:

• The firewall log shows an Allow row for either login.microsoftonline.com or login.microsoft.com from the workload subnet source IP during the failure window, AND the failure persists — the network path is not the controlling variable.
• The workload subnet has no UDR forcing egress through Azure Firewall (no 0.0.0.0/0 → VirtualAppliance next-hop) — there is no firewall on the path.
• Direct probes from the same egress path to both https://login.microsoftonline.com/<tenant>/.well-known/openid-configuration and https://login.microsoft.com/<tenant>/.well-known/openid-configuration (for example via az containerapp exec running curl against both hosts) succeed and return discovery JSON. Probing a single Entra authority host is not sufficient to disprove H4: the OIDC discovery client picks one host at runtime, so a firewall rule that permits only one FQDN will silently drop discovery requests that happen to pick the other, and a one-host probe cannot distinguish that condition from a fully-open path.
• The failure signature is an HTTP 401 or 403 from Microsoft Entra ID or from Key Vault rather than a transport-level EOF on the OIDC discovery URL. HTTP status codes point to H2 (Key Vault RBAC) or a token audience / SDK config issue, not to a network path drop.

No matching row is not disproof on its own. The absence of any Allow or Deny row for the Entra authority FQDNs in the query result can also mean: firewall diagnostic logging is not enabled, diagnostics are configured to a different Log Analytics workspace, the query window is outside the ingestion latency window (ingestion typically takes 60 seconds or more), the source IP filter (ipv4_is_in_range(SourceIp, AcaSubnet)) does not match the actual egress subnet CIDR, or the query is targeting the wrong tenant's diagnostics table. Before treating an empty result as a signal, confirm at least one diagnostic setting routes the AzureFirewallApplicationRule log category to the queried Log Analytics workspace, confirm whether the workspace uses resource-specific tables (which surface rows in the AZFWApplicationRule table) or Azure diagnostics mode (which surface rows in the AzureDiagnostics table with Category == "AzureFirewallApplicationRule") or both, and confirm the query window extends at least five minutes past the failure timestamp.

H4 Variants: When Base H4 KQL Returns Zero Rows¶

The base H4 hypothesis assumes a specific topology: hub-spoke networking + 0.0.0.0/0 UDR forcing egress through Azure Firewall Basic/Standard + missing Application Rule + AzFW diagnostic settings enabled + Deny row visible in AZFWApplicationRule. The reproducer lab reproduces this exact scenario end-to-end.

However, the same failure surface (Unable to get value using Managed identity + .well-known/openid-configuration: EOF on either Entra authority FQDN) can result from six other topology variants where the base H4 KQL query returns zero rows. Before treating an empty KQL result as evidence against H4, work through the preflight and variants table below.

Preflight (must pass before treating zero rows as disproof):

• At least one diagnostic setting on the Firewall resource routes the AzureFirewallApplicationRule log category to the queried Log Analytics workspace — verify with az monitor diagnostic-settings list --resource $FIREWALL_ID. AzureFirewallApplicationRule is the log-category name; the actual destination table depends on the diagnostic setting's delivery mode (resource-specific mode produces rows in the AZFWApplicationRule table; Azure diagnostics mode produces rows in the AzureDiagnostics table with Category == "AzureFirewallApplicationRule"). Query the mode the diagnostic setting actually uses.
• The Log Analytics workspace queried matches the workspace configured in the Firewall's diagnostic settings target.
• The query window (StartTime, EndTime) extends at least five minutes past the failure timestamp to account for ingestion latency.
• The ipv4_is_in_range(SourceIp, AcaSubnet) filter uses the actual workload subnet CIDR — verify with az containerapp env show --query "properties.vnetConfiguration.infrastructureSubnetId" and inspect the corresponding subnet's addressPrefix.
• The query targets the correct tenant and subscription context.

If preflight passes and the query still returns zero rows for the failure window, the failure is likely a topology variant. Work through the table in the rule-out order at the bottom.

Rule-out order (cheapest verification first):

1. H4a — base H4 KQL. If Deny row present → confirmed, apply Section 8 Step 5 mitigation.
2. H4b — one CLI call to check diagnostic settings. If diagnostic categories are disabled, enable them and re-reproduce.
3. H4e — one az containerapp exec call for nslookup. Fast to rule out DNS override.
4. H4c — NSG rule enumeration. One CLI call pair (az network vnet subnet show → az network nsg rule list) is enough to rule in or out the documented AzureActiveDirectory:443 deny pattern; flow logs are optional corroboration only.
5. H4d — multiple az network vhub calls plus understanding of Virtual WAN topology. Requires the customer environment to have Virtual WAN deployed.
6. H4g — TLS inspection scan. Requires understanding of the AzFW Premium or NVA TLS inspection configuration.
7. H4f — NVA vendor-specific tooling. Escalate to the NVA vendor.

Customer scenario mapping:

• "Virtual WAN Hub + Firewall + Routing Intent" + "customer reports no firewall block logs" → most commonly H4d combined with H4b, and in customer environments that layer additional egress controls, potentially also H4g (Premium TLS inspection on the Entra authority) or H4e (custom DNS override on the workload VNet). Verify Routing Intent policy state, Firewall diagnostic settings, TLS-inspection policy scope, and workload-VNet DNS configuration in parallel. If Routing Intent effective routes look correct AND diagnostic settings are enabled AND KQL still returns zero rows for the failure window, treat the empty result as pointing to a route-convergence gap or an interception path rather than a missing Application Rule — re-check effective routes after the policy's provisioning state reaches Succeeded, and inspect TLS-inspection exclusions and the workload VNet's dhcpOptions.dnsServers before concluding the network path is not the controlling variable.
• "3rd-party NVA in path" + "no AzFW logs" → H4f. Engage the NVA vendor with the failure timestamp and destination FQDN list.
• "AzFW logs show TLS handshake errors on Entra authority FQDN" → H4g. Exempt the Entra authority FQDNs (login.microsoftonline.com and login.microsoft.com) from TLS inspection at the intercepting device. Do not attempt to fix this by installing a corporate root CA into the customer workload image — the failing caller is the ACA-managed control-plane worker, not the customer's container, and the workload image's trust store does not gate that path.
• "OIDC discovery succeeds sometimes, fails sometimes" → intermittent behavior — verify the OIDC discovery client host selection (either login.microsoftonline.com or login.microsoft.com) and confirm the firewall rule permits both FQDNs. This is the base H4a case with a partial rule, not a topology variant.

The H4b (logging-gap) variant now has a fully reproducible end-to-end lab — see the ACA Secret Key Vault Reference — Logging Gap Variant (H4b) Lab, which proves that re-enabling the AzureFirewallApplicationRule diagnostic category restores observability of the denial without restoring connectivity. The H4e (custom DNS override) variant also has a fully reproducible end-to-end lab — see the ACA Secret Key Vault Reference — DNS Override Variant (H4e) Lab, which proves that a custom Private DNS zone overriding the Entra authority record breaks OIDC discovery before egress reaches the firewall, and that removing the override restores connectivity. The H4d (Virtual WAN + Routing Intent) variant also has a fully reproducible end-to-end lab — see the ACA Secret Key Vault Reference — Virtual WAN Routing Intent Variant (H4d) Lab, which proves that enabling Routing Intent to route the ACA infrastructure VNet's egress through the Virtual WAN Hub firewall breaks Entra authority reachability, and that disabling Routing Intent — with the firewall policy left unchanged — restores connectivity. The H4c (NSG deny) variant also has a fully reproducible end-to-end lab — see the ACA Secret Key Vault Reference — NSG Deny Variant (H4c) Lab, which proves that an outbound NSG rule denying AzureActiveDirectory:443 on the ACA workload subnet breaks Entra authority reachability before any firewall path, and that adding the documented higher-priority Allow restores connectivity. The H4g (Azure Firewall Premium TLS inspection) variant also has a fully reproducible end-to-end lab — see the ACA Secret Key Vault Reference — Firewall Premium TLS Inspection Variant (H4g) Lab, which proves that enabling TLS inspection on the Entra authority FQDNs breaks the control-plane's OIDC discovery call, and that exempting those FQDNs from TLS inspection — with all other topology unchanged — restores connectivity. The H4f (3rd-party NVA) variant also has a fully reproducible end-to-end lab — see the ACA Secret Key Vault Reference — Third-Party NVA Surrogate Variant (H4f) Lab, which substitutes a Linux forwarding VM for a vendor NVA and proves that a single forwarding-plane DROP rule for the AzureActiveDirectory service-tag prefixes on tcp/443 breaks Entra authority reachability — with no Azure Firewall and no TLS inspection anywhere in the topology — and that removing the same rule restores connectivity. With H4f complete, all six H4 network-path variants (H4b, H4c, H4d, H4e, H4f, H4g) now have end-to-end reproducer labs; the shared tracking issue is issue #307.

For a fully reproducible end-to-end proof of this hypothesis — including an offline verifier that re-validates the reader-generated H0 → H1 → H2 cohort against 17 gates and enumerates the 8 documented explicit drops (stderr wording, log ingestion latency, retry cadence, component identity, response body shape, token caching, SKU generality, region generality) — see the ACA Secret Key Vault Reference — Managed Identity Network Path Lab.

7. Likely Root Cause Patterns¶

8. Immediate Mitigations¶

1. Ensure all secretRef values map to existing secrets.
2. For Key Vault references, confirm managed identity and vault RBAC/policy access.
3. Rotate or set secret values and deploy a new revision.
4. Validate app behavior with expected config value present.

5. If the workload subnet routes egress through Azure Firewall (or another NVA), verify the firewall policy contains an Application Rule permitting outbound HTTPS to both login.microsoftonline.com and login.microsoft.com from the workload subnet, and confirm the firewall log shows an Allow row for either Entra authority FQDN in the failure window. If the rule is missing, add or restore it:

   Command

   Why it is used

   az network firewall policy rule-collection-group collection add-filter-collection ...

   Adds an ApplicationRule collection to the existing firewall policy rule-collection group that permits outbound HTTPS to both login.microsoftonline.com and login.microsoft.com from the workload subnet CIDR. Both FQDNs are listed because the managed-identity OIDC discovery client picks one Entra authority host at runtime; a rule that names only one FQDN will silently drop discovery attempts that happen to pick the other host.

   az network firewall policy rule-collection-group collection add-filter-collection \
       --resource-group "$RG" \
       --policy-name "$FIREWALL_POLICY_NAME" \
       --rule-collection-group-name "$FIREWALL_POLICY_RCG_NAME" \
       --name "aca-entra-authority-allow" \
       --collection-priority 200 \
       --action Allow \
       --rule-type ApplicationRule \
       --rule-name "allow-entra-authority" \
       --source-addresses "$ACA_WORKLOAD_SUBNET_CIDR" \
       --protocols Https=443 \
       --target-fqdns "login.microsoftonline.com" "login.microsoft.com"
   

   Command

   Purpose

   az network firewall policy rule-collection-group collection add-filter-collection --resource-group "$RG" --policy-name "$FIREWALL_POLICY_NAME" --rule-collection-group-name "$FIREWALL_POLICY_RCG_NAME" --name "aca-entra-authority-allow" --collection-priority 200 --action Allow --rule-type ApplicationRule --rule-name "allow-entra-authority" --source-addresses "$ACA_WORKLOAD_SUBNET_CIDR" --protocols Https=443 --target-fqdns "login.microsoftonline.com" "login.microsoft.com"

   Adds the firewall application rule that permits both Entra authority FQDNs from the workload subnet, which is the direct mitigation when OIDC discovery is being dropped on the egress path.

6. If Step 5 confirms the firewall Application Rule for both Entra authority FQDNs is already present AND the base H4 KQL returns zero rows in the failure window, work through the H4 Variants table in rule-out order. Cheapest verification first: check AzFW diagnostic settings enabled at the Firewall resource (H4b, one CLI call), then verify DNS resolution of the Entra authority hosts from the workload path (H4e, one az containerapp exec call for nslookup plus a check of the workload VNet's dhcpOptions.dnsServers), then inspect the outbound NSG rules on the workload subnet for a Deny on 443/tcp to AzureActiveDirectory or Entra IP ranges (H4c), then verify Virtual WAN Hub Routing Intent and effective routes if the customer environment uses Virtual WAN (H4d), then verify whether outbound TLS to the Entra authority is being intercepted by running openssl s_client from a workload replica and inspecting the returned certificate chain (H4g — if intercepted, exempt the Entra authority FQDNs from TLS inspection at the intercepting device rather than modifying the workload image), and finally 3rd-party NVA policy if the customer environment has a vendor NVA in path (H4f). The customer scenario "Virtual WAN Hub + Firewall + Routing Intent + no firewall block logs" typically resolves as H4d combined with H4b — verify Routing Intent state and Firewall diagnostic settings in parallel, and if effective routes look correct but the base H4 KQL is still empty, re-check the effective routes after the routing policy's provisioning state reaches Succeeded before concluding the route topology is not the controlling variable.

9. Prevention¶

• Standardize secret naming and reference patterns.
• Add secret existence checks in deployment pipelines.
• Schedule regular Key Vault permission audits.

See Also¶

• Managed Identity Auth Failure
• Revision Provisioning Failure
• Secret Reference Failures KQL
• ACA Secret Key Vault Reference — Managed Identity Network Path Lab
• Egress Control — Required Outbound Dependencies

Sources¶

• Manage secrets in Azure Container Apps
• Use managed identity to authenticate to Azure Key Vault from Azure Container Apps
• Managed identities in Azure Container Apps
• User-defined routes with Azure Container Apps
• Configure a firewall in Azure Container Apps environments