Skip to content

Image Pull Fails in Restricted Egress

Hypothesis triage for image pulls under restricted egress:

flowchart TD
    S["Pods in ImagePullBackOff under restricted egress"] --> H1{"Image reference or pull secret broken?"}
    H1 -- yes --> R1["Fix image ref, tag, or pull secret"]
    H1 -- no --> H2{"ACR private endpoint DNS resolves publicly?"}
    H2 -- yes --> R2["Fix private DNS zone linkage"]
    H2 -- no --> H3{"Firewall, UDR, or proxy blocks registry endpoints?"}
    H3 -- yes --> R3["Allow ACR and MCR FQDNs on the egress path"]
    H3 -- no --> H4{"Proxy intercepts TLS or misses no-proxy exclusions?"}
    H4 -- yes --> R4["Correct TLS handling and bypass CIDRs"]
    H4 -- no --> EV["Diagnose: prove image vs network first, confirm egress model"]
    EV --> RES["Resolution per matched cause"]
    RES --> PREV["Prevention: validate egress rules with each cluster change"]

Symptom

Pods stay in ImagePullBackOff or ErrImagePull after deployment in a cluster that uses restricted egress, forced tunneling, Azure Firewall, proxy routing, or private endpoints.

Possible Causes

  • Wrong image reference, missing tag, or broken pull secret.
  • Azure Container Registry private endpoint exists, but private DNS is missing or still resolves publicly.
  • Firewall, UDR, or proxy blocks required ACR, Microsoft Container Registry, or other AKS dependency endpoints.
  • Required mcr.microsoft.com or related artifact FQDNs are not allowed.
  • Proxy interception blocks TLS or misses the correct no-proxy exclusions.
  • Private endpoints, pod CIDRs, or service CIDRs are not bypassed correctly in the proxy path.

Diagnosis Steps

  1. Prove whether the failure is an image or authentication problem before chasing networking.

    kubectl describe pod <pod-name> --namespace <namespace>
kubectl get events --sort-by=.lastTimestamp

  2. Confirm the cluster egress model and networking plugin.

    az aks show \
    --resource-group "$RG" \
    --name "$CLUSTER_NAME" \
    --query "{outboundType:networkProfile.outboundType,networkPlugin:networkProfile.networkPlugin}" \
    --output yaml
    Command Purpose
    az aks show Show the outbound type and network plugin.
    --resource-group Resource group that contains the AKS cluster.
    --name Name of the AKS cluster.
    --query Selects the outbound type and network plugin.
    --output Output format for the result.

  3. Inspect the registry exposure model.

    az acr show \
    --resource-group "$ACR_RG" \
    --name "$ACR_NAME" \
    --query "{loginServer:loginServer,publicNetworkAccess:publicNetworkAccess}" \
    --output yaml

az network private-endpoint-connection list \
    --id "/subscriptions/<subscription-id>/resourceGroups/$ACR_RG/providers/Microsoft.ContainerRegistry/registries/$ACR_NAME" \
    --output table
    Command Purpose
    az acr show Show the registry login server and public access.
    --resource-group Resource group that contains the registry.
    --name Name of the container registry.
    --query Selects the login server and public network access.
    --output Output format for the result.
    az network private-endpoint-connection list List private endpoint connections on the registry.
    --id Resource ID of the container registry.
    --output Output format for the result.

  4. Check whether core platform pods show a wider egress problem.

    kubectl get pods --namespace kube-system --output wide

  5. Compare the failing image hostname with the allowed FQDNs and any proxy bypass list for private endpoints, pod CIDRs, and service CIDRs.

Resolution

  • Fix the image name, tag, or pull secret first when the error is authentication- or manifest-related.
  • Add or repair the ACR private endpoint and its private DNS zone links when the registry should stay private.
  • Allow the required AKS, MCR, and ACR FQDNs through firewall, UDR, or proxy controls.
  • Add proxy bypass entries for private endpoints, pod CIDRs, service CIDRs, and registry private DNS names.
  • Prefer ACR private endpoints for restricted-egress clusters rather than relying on public registry paths.

Prevention

  • Standardize registry patterns for restricted-egress clusters before onboarding workloads.
  • Keep the required outbound dependency list under change control with the network team.
  • Validate proxy no-proxy coverage whenever private endpoints or CIDR ranges change.
  • Treat image pull failures as both application onboarding checks and platform egress checks.

See Also

Sources