Skip to content

SNAT Port Exhaustion

Symptom

Outbound calls fail intermittently, image pulls or external API calls time out under load, and retries succeed later without a clear application change.

Possible Causes

  • SNAT ports are exhausted on Standard Load Balancer outbound rules.
  • NAT Gateway is undersized or attached to the wrong subnet.
  • Applications create excessive short-lived outbound connections.
  • Egress firewall or NVA is the real bottleneck rather than AKS SNAT.
  • DNS, proxy, or upstream throttling is being misread as SNAT exhaustion.
  • Missing private endpoints force unnecessary public SNAT use.

Diagnosis Steps

This playbook covers SNAT concepts, symptoms, evidence collection, and mitigation decisions. The Ingress and Networking Metrics reference remains authoritative for SNAT metric definitions and tables, including snatConnectionsCount, allocatedSnatPorts, usedSnatPorts, and SNAT capacity notes — consult it there instead of duplicating those tables here.

flowchart TD
    A[Intermittent outbound failures] --> B[Identify outbound type]
    B --> C{loadBalancer or NAT Gateway?}
    C -->|loadBalancer| D[Inspect outbound rules and SNAT pressure]
    C -->|NAT Gateway| E[Confirm NAT Gateway attachment and scale]
    D --> F[Check app connection patterns]
    E --> F
    F --> G[Disprove firewall, proxy, DNS, or upstream bottlenecks]
    G --> H[Reduce public SNAT demand with private endpoints or redesign]

  1. Identify the outbound configuration the cluster is actually using.

    az aks show \
    --resource-group "$RG" \
    --name "$CLUSTER_NAME" \
    --query "{outboundType:networkProfile.outboundType,loadBalancerProfile:networkProfile.loadBalancerProfile,natGatewayProfile:networkProfile.natGatewayProfile}" \
    --output yaml
    Command Purpose
    az aks show Show the outbound type and load balancer profiles.
    --resource-group Resource group that contains the AKS cluster.
    --name Name of the AKS cluster.
    --query Selects outbound type, load balancer, and NAT gateway profiles.
    --output Output format for the result.

  2. Collect cluster events and workload-side symptoms from the same time window.

    kubectl get events --all-namespaces --sort-by=.lastTimestamp
kubectl logs <pod-name> --namespace <namespace> --since=30m

  3. If loadBalancer is in use, inspect the outbound rule configuration.

    az network lb outbound-rule list \
    --resource-group "$NODE_RG" \
    --lb-name "$LOAD_BALANCER_NAME" \
    --output table
    Command Purpose
    az network lb outbound-rule list List outbound rules on the load balancer.
    --resource-group Node resource group that contains the load balancer.
    --lb-name Name of the load balancer.
    --output Output format for the result.

  4. If NAT Gateway is in use, confirm the expected gateway and subnet attachment.

    az network nat gateway show \
    --resource-group "$RG" \
    --name "$NAT_GATEWAY_NAME" \
    --output json
    Command Purpose
    az network nat gateway show Show the NAT gateway used for egress.
    --resource-group Resource group that contains the NAT gateway.
    --name Name of the NAT gateway.
    --output Output format for the result.

  5. Pull the relevant Azure Monitor metrics for the suspected egress resource and compare them with the metric reference page instead of recreating local metric tables.

    az monitor metrics list \
    --resource "$RESOURCE_ID" \
    --metric "SnatConnectionCount" \
    --interval PT5M
    Command Purpose
    az monitor metrics list List SNAT connection metrics for the resource.
    --resource Resource ID to query.
    --metric Metric to retrieve.
    --interval Aggregation interval for the metric.

Resolution

  • Move from loadBalancer to NAT Gateway when SNAT scale needs are consistently above Standard Load Balancer comfort levels.
  • Add outbound IP capacity where the chosen egress design supports it.
  • Reduce short-lived outbound connections with client-side pooling and keep-alives.
  • Use private endpoints to reduce avoidable public SNAT demand.
  • If UDR sends egress through a firewall or NVA, scale and monitor that device as the true bottleneck.

Prevention

  • Choose the outbound type based on expected concurrency, not only initial simplicity.
  • Review connection pooling behavior during workload onboarding.
  • Keep SNAT metrics and incident runbooks aligned with the shared metrics reference.
  • Prefer private endpoints for heavy platform dependencies when restricted-egress design allows them.

See Also

Sources