Private-endpoint DNS failure¶

This lab documents the first Azure Storage troubleshooting substrate in the repository: a storage account with blob private access, a blob private endpoint, a privatelink.blob.core.windows.net zone, a VNet link, and a client VM that tests name resolution from the private path. The failure model is deliberately narrow: delete the private DNS VNet link, prove that blob resolution no longer returns the private endpoint IP, then restore the link and falsify the failure hypothesis.

Lab Metadata¶

1) Background¶

The substrate in labs/private-endpoint-dns-failure/ provisions these components:

• A StorageV2 account with publicNetworkAccess disabled and blob access exposed through a private endpoint.
• A virtual network with a client subnet and a dedicated private-endpoint subnet.
• A privatelink.blob.core.windows.net private DNS zone plus a VNet link named blob-zone-link.
• A Linux VM that installs dnsutils and curl so the lab can run nslookup, dig, and HTTPS reachability checks from the same private path that an application would use.

In the healthy state, ${STORAGE_NAME}.blob.core.windows.net should resolve on the client VM to the private endpoint IP that Azure assigned to the blob private endpoint. The substrate's fault injection step removes the only DNS link that lets the client VNet resolve the privatelink zone, so the same lookup should fall back to a public or otherwise non-private answer.

mermaid flowchart TD A[Deploy substrate with blob private endpoint] --> B[Validate blob FQDN resolves to PE IP] B --> C[Delete blob-zone-link from privatelink.blob.core.windows.net] C --> D[Re-run nslookup and dig from client VM] D --> E{Answer still matches PE IP?} E -->|Yes| F[Hypothesis not proven yet] E -->|No| G[DNS failure reproduced] G --> H[Re-create VNet link] H --> I[Re-run name resolution and HTTPS reachability] I --> J[Private answer returns and access path recovers]

2) Hypothesis¶

If the privatelink.blob.core.windows.net VNet link is deleted, then the client VM will stop resolving ${STORAGE_NAME}.blob.core.windows.net to the private endpoint IP. Because the storage account disables public network access, requests that no longer follow the private path will fail until the private DNS VNet link is restored.

Predictions for a future live run:

• Before fault injection, nslookup and dig from the client VM return the blob private endpoint IP.
• After the VNet link is deleted, the same commands no longer return that private IP.
• After the VNet link is recreated, the blob FQDN again resolves to the private endpoint IP and HTTPS requests once again reach the storage service endpoint.

3) Runbook¶

This page is documentation-only. Do not treat the commands below as already executed in this PR; they are the exact future live-run sequence grounded in the substrate.

Deploy the healthy baseline¶

Use the substrate exactly as authored. Set INJECT_FAULT="false" so you can measure the healthy state before deleting the DNS link.

export RG="rg-storage-pe-dns-lab"
export LOCATION="koreacentral"
export INJECT_FAULT="false"

bash labs/private-endpoint-dns-failure/scripts/reproduce.sh

Record the deployed names¶

export DEPLOYMENT_NAME="private-endpoint-dns-failure"

export STORAGE_NAME="$(az deployment group show \
    --resource-group "$RG" \
    --name "$DEPLOYMENT_NAME" \
    --query properties.outputs.storageAccountName.value \
    --output tsv)"

export VM_NAME="$(az deployment group show \
    --resource-group "$RG" \
    --name "$DEPLOYMENT_NAME" \
    --query properties.outputs.clientVmName.value \
    --output tsv)"

Inject the DNS fault¶

export ZONE_LINK_NAME="blob-zone-link"

az network private-dns link vnet delete \
    --resource-group "$RG" \
    --zone-name privatelink.blob.core.windows.net \
    --name "$ZONE_LINK_NAME" \
    --yes

Restore the DNS path¶

export VNET_NAME="$(az deployment group show \
    --resource-group "$RG" \
    --name "$DEPLOYMENT_NAME" \
    --query properties.outputs.clientVnetName.value \
    --output tsv)"

az network private-dns link vnet create \
    --resource-group "$RG" \
    --zone-name privatelink.blob.core.windows.net \
    --name "$ZONE_LINK_NAME" \
    --virtual-network "$VNET_NAME" \
    --registration-enabled false

4) Experiment Log¶

This PR does not include a live Azure run. The log below captures the intended experiment phases and the evidence standard for the later validation pass.

1. Healthy baseline
   • [Not Proven] Deploy the substrate with INJECT_FAULT="false".
   • [Not Proven] Capture the blob private endpoint IP, the VNet link state, and client-VM nslookup / dig output proving the blob FQDN resolves privately before fault injection.
2. Faulted state
   • [Not Proven] Delete blob-zone-link from privatelink.blob.core.windows.net.
   • [Not Proven] Re-run the same queries from the client VM and confirm that the blob FQDN no longer resolves to the private endpoint IP.
3. Post-fix falsification
   • [Not Proven] Re-create blob-zone-link.
   • [Not Proven] Re-run the exact same verification set and confirm the blob FQDN again resolves to the private endpoint IP.
   • [Not Proven] Treat the hypothesis as falsified only when the same client path that failed during the faulted state now resolves and reaches the private endpoint correctly.

5) Verification Queries¶

Use the same checks three times during a future live run: healthy baseline, faulted state, and post-fix falsification. Record the private endpoint IP first, then compare every later lookup against that baseline value.

Query set A: Capture the authoritative private endpoint IP and DNS-link state¶

az network private-endpoint show \
    --resource-group "$RG" \
    --name pednslab-blob-pe \
    --query 'customDnsConfigs[].ipAddresses[]' \
    --output tsv

az network private-dns link vnet show \
    --resource-group "$RG" \
    --zone-name privatelink.blob.core.windows.net \
    --name "$ZONE_LINK_NAME" \
    --output jsonc

Query set B: Prove name resolution from the client VM¶

az vm run-command invoke \
    --resource-group "$RG" \
    --name "$VM_NAME" \
    --command-id RunShellScript \
    --scripts "nslookup ${STORAGE_NAME}.blob.core.windows.net"

az vm run-command invoke \
    --resource-group "$RG" \
    --name "$VM_NAME" \
    --command-id RunShellScript \
    --scripts "dig +short ${STORAGE_NAME}.blob.core.windows.net"

Query set C: Prove the data path recovers after the DNS fix¶

az vm run-command invoke \
    --resource-group "$RG" \
    --name "$VM_NAME" \
    --command-id RunShellScript \
    --scripts "curl -I --max-time 10 https://${STORAGE_NAME}.blob.core.windows.net/"

Query set D: Correlate the fault and fix in Azure Activity¶

AzureActivity
| where ResourceGroup == "<resource-group>"
| where OperationNameValue in (
    "Microsoft.Network/privateDnsZones/virtualNetworkLinks/delete",
    "Microsoft.Network/privateDnsZones/virtualNetworkLinks/write"
)
| where ResourceProviderValue == "MICROSOFT.NETWORK"
| project TimeGenerated, OperationNameValue, ActivityStatusValue, Resource
| order by TimeGenerated asc

Pass / fail rules¶

6) Portal Evidence¶

Status: pending live capture.

When this lab is executed against a real Azure environment, capture Portal evidence into docs/assets/troubleshooting/private-endpoint-dns-failure/ and verify the final rendered images before merge.

Recommended captures for the live follow-up:

• Storage account Networking blade showing public network access disabled and the blob private endpoint connection present.
• Private endpoint Overview blade showing the approved blob private endpoint and its private IP.
• Private DNS zone Virtual network links blade before fault injection, after deletion, and after the link is recreated.
• VM Run command or equivalent evidence proving the client-side nslookup result before fault injection and after the fix.

Do not reference image files from this page until those captures actually exist.

Clean Up¶

export RG="rg-storage-pe-dns-lab"

bash labs/private-endpoint-dns-failure/scripts/cleanup.sh

Related Playbook¶

• Private Endpoint and DNS Issues

See Also¶

• Lab Guides
• Troubleshooting Home
• Cannot Access Storage Account
• Use Private Endpoints

Sources¶

• Use private endpoints for Azure Storage
• Azure Private Endpoint private DNS zone values
• Troubleshoot Azure Private Endpoint connectivity problems