Lab: Private Endpoint Route Fault with Correct DNS¶

This lab reproduces hypothesis H3 of the Private Endpoint / Custom DNS / Route Confusion playbook: the storage name resolves to the private endpoint address, the endpoint is approved, and calls still fail because a user-defined route sends the traffic somewhere else. DNS is held correct in every phase, so the route is the only variable.

Lab Metadata¶

1) Background¶

A private endpoint call depends on three separate layers: the endpoint (provisioned and approved), DNS (the name must resolve to the endpoint address), and the network path (packets from the integration subnet must reach that address). The DNS lab breaks the DNS layer. This lab keeps DNS and the endpoint healthy and breaks only the path.

A private endpoint adds a /32 system route for its address. Microsoft Learn documents that a user-defined route overrides that route only when private endpoint network policies for route tables are enabled on the endpoint subnet, and only when its prefix is no broader than the virtual network address space; a 0.0.0.0/0 route does not override it. This lab uses an exact /32 route on the integration subnet, pointing at an unused address in an empty subnet, so the next hop never answers.

flowchart TD
    A[Healthy: name resolves to the endpoint address] --> B[Call reaches the endpoint, fast 409]
    B --> C[Add /32 route for the endpoint address to an unused appliance address]
    C --> D[Name still resolves to the endpoint address]
    D --> E[TCP connect times out after 8 s, ResultCode 0]
    E --> F[Delete the route]
    F --> G[Fast 409 again with the same DNS answer]

The environment deployed by main.bicep:

• Linux App Service plan (B1) with a Python app integrated with snet-<base>-int, which has the route table attached (empty at deployment).
• Storage account with public network access disabled and a blob private endpoint in snet-<base>-pep, where private endpoint network policies are Enabled.
• privatelink.blob.core.windows.net private DNS zone, linked to the VNet, with the endpoint's A record through a zone group.
• Empty snet-<base>-nva subnet that provides the unused next-hop address 10.50.3.4.
• Log Analytics workspace with App Service diagnostic logs, and workspace-based Application Insights, which records each storage call in AppDependencies.

The app exposes /resolve (resolver answers for the standard and privatelink names) and /connect (one anonymous ?comp=list call to each name with an 8-second timeout). /connect always answers 200; the outcome is in its JSON body.

2) Hypothesis¶

Setup¶

Deploy main.bicep, deploy the app, and confirm /resolve returns the private endpoint address for the storage name before changing anything.

Hypothesis¶

If a /32 user-defined route on the integration subnet sends the private endpoint address to a next hop that does not forward traffic, calls to the storage account time out at the TCP connect stage while the storage name keeps resolving to the private endpoint address, and removing the route restores the calls without any DNS change.

3) Runbook¶

Experiment¶

export RG="rg-lab-pe-route"
export LOCATION="koreacentral"

az group create --name "$RG" --location "$LOCATION"
az deployment group create \
    --resource-group "$RG" \
    --template-file labs/private-endpoint-route-fault/main.bicep \
    --parameters baseName=labpe

Package the app/ directory as a ZIP and deploy it:

export APP_NAME=$(az webapp list --resource-group "$RG" --query "[0].name" --output tsv)
az webapp deploy --resource-group "$RG" --name "$APP_NAME" --src-path app.zip --type zip

Run three phases, probing /resolve and /connect about every 15 seconds in each:

1. Healthy: probe without changes.
2. Fault: bash labs/private-endpoint-route-fault/trigger.sh "$RG" fault, wait about 3 minutes, then probe.
3. Recovery: bash labs/private-endpoint-route-fault/trigger.sh "$RG" restore, wait about 3 minutes, then probe.

After each phase, run bash labs/private-endpoint-route-fault/verify.sh "$RG". It prints whether the storage name still resolves to the endpoint address and the most recent AppDependencies result codes for the standard FQDN.

4) Experiment Log¶

Execution¶

The lab ran on 2026-10-02 in Korea Central with one B1 instance. The healthy phase started at 17:29 UTC. The route was added at 17:31 UTC, and the fault phase probed 17:34-17:38 UTC. The route was deleted at 17:38 UTC, and the recovery phase probed 17:41-17:43 UTC. Each phase made 8 /connect and 8 /resolve calls. Endpoint state was Succeeded / Approved and the zone link state Completed throughout.

Observation¶

Measurement¶

Analysis¶

The DNS answer, the endpoint approval, and the zone link were identical across the three phases; the only change was the /32 route. The calls failed when the route was added and recovered when it was removed. That A/B/A sequence isolates the route as the cause.

The fault does not show up in the signals most dashboards watch: inbound status stayed 200, there were no 5xx, and the console log was empty. The signals that changed were dependency duration rising to the client timeout and ResultCode changing from 409 to 0.

Conclusion¶

The hypothesis is supported: with DNS and the endpoint healthy, a /32 user-defined route to a non-forwarding next hop turned fast storage responses into 8-second connect timeouts, and deleting the route restored them.

Falsification¶

The recovery phase is the falsification step. If anything other than the route had caused the timeouts (DNS cache, endpoint state, storage throttling), deleting the route would not have restored the 409 responses. They returned within the 3-minute wait, with the same DNS answer as before.

Evidence¶

The lab's verify.sh reproduces the two decisive checks: the resolver answer compared with the endpoint NIC address, and the newest AppDependencies result codes. Raw run output stayed outside the repository because it contains resource names and addresses.

Solution¶

Remove or correct the user-defined route so traffic to the private endpoint address uses the endpoint's /32 system route or a next hop that forwards it (for example, a firewall that allows the flow).

Prevention¶

• When route tables are attached to an integration subnet, list every route whose prefix covers private endpoint addresses before changing it.
• Alert on dependency duration and ResultCode 0 for private dependencies, not only on inbound 5xx.
• When forcing private endpoint traffic through an appliance, confirm the appliance forwards the flow before applying the route.

Takeaway¶

Correct DNS does not prove a working private path. When the name resolves to the endpoint address and calls still time out, check routes on the integration subnet before changing DNS.

Support Takeaway¶

Ask for the resolver answer and the dependency result code together. A private endpoint address with ResultCode 0 at the client timeout points at the path layer (route or NSG). A public address points at DNS. A fast HTTP status shows the endpoint was reached.

Expected Evidence¶

Clean Up¶

az group delete --name "$RG" --yes --no-wait

Related Playbook¶

• Private Endpoint / Custom DNS / Route Confusion

See Also¶

• Lab: DNS Resolution (VNet)
• DNS Resolution with VNet-Integrated App Service
• Outbound network first-10-minutes checklist

Sources¶

• Manage network policies for private endpoints
• Virtual network traffic routing
• Azure private endpoint DNS configuration
• Integrate your app with an Azure virtual network