Skip to content

CoreDNS Query Latency or Drops

Symptom

Pods intermittently time out on DNS lookups, application logs show repeated resolver latency or missing responses, or cluster-wide DNS dashboards show growing failure or retry patterns.

Typical application signatures include:

  • i/o timeout
  • no such host after intermittent success
  • Temporary failure in name resolution
  • HTTP or gRPC client errors that disappear when retried

Possible Causes

  • CoreDNS pods are CPU-, memory-, or node-pressure bound.
  • A recent coredns-custom override introduced a slow or broken forward path.
  • Upstream VNet DNS or custom DNS servers are slow.
  • The cluster has enough DNS volume that centralized CoreDNS is now a bottleneck.
  • Network policy blocks DNS traffic to CoreDNS or LocalDNS.

Diagnosis Steps

flowchart TD
    A[DNS latency or dropped lookups] --> B[Check app errors and namespace blast radius]
    B --> C[Inspect CoreDNS pods and restart history]
    C --> D{Centralized CoreDNS issue or upstream resolver issue?}
    D -->|CoreDNS concentration| E[Review CoreDNS load and LocalDNS decision]
    D -->|Upstream resolver path| F[Review forwarders VNet DNS and custom DNS]
    E --> G[Validate policy and rollout a fix]
    F --> G

  1. Confirm whether the issue is namespace-specific or cluster-wide.

    kubectl get pods \
    --all-namespaces \
    --output wide

  2. Inspect CoreDNS pod health, restarts, and scheduling location.

    kubectl get pods \
    --namespace kube-system \
    --selector k8s-app=kube-dns \
    --output wide

  3. Describe a CoreDNS pod to inspect events.

    kubectl describe pod <coredns-pod-name> \
    --namespace kube-system

  4. Review current CoreDNS and custom configuration.

    kubectl get configmaps \
    --namespace kube-system \
    coredns coredns-custom \
    --output yaml

  5. If ACNS observability is enabled, check the DNS dashboards called out by Microsoft Learn.

    • DNS (Cluster) for node or cluster concentration.
    • DNS (Workload) for whether the failing workload is only one deployment or a platform-wide issue.

  6. If the cluster does not use LocalDNS, ask whether the symptoms match centralized DNS concentration rather than a broken record or broken upstream resolver.

Resolution

  • Roll back or correct a bad coredns-custom forwarder or rewrite.
  • If the issue is upstream, fix the VNet DNS or custom DNS resolver rather than only restarting CoreDNS.
  • If CoreDNS itself is the concentration point, add headroom and evaluate LocalDNS.
  • If policy blocks DNS traffic, add the required allow rules for CoreDNS or LocalDNS destinations.

Prevention

  • Treat every CoreDNS customization as a production change with rollback notes.
  • Baseline CoreDNS pod health and restart rate before incidents.
  • Prefer LocalDNS when high query volume or conntrack pressure is a recurring pattern.
  • Keep explicit DNS allow rules in namespace default-deny policies.

See Also

Sources