When ndots turns service discovery into a DNS storm

Measure how Kubernetes search paths and ndots settings multiply internal lookups, saturate CoreDNS, and create latency spikes before application traffic even begins.

Introduction to Kubernetes DNS Architecture

Kubernetes provides service discovery DNS primarily through the kube-dns add‑on, which in modern clusters is almost always a CoreDNS deployment. Each pod receives an /etc/resolv.conf that points to the cluster‑internal DNS service IP (typically 10.96.0.10 or the kube‑dns Service IP). The file contains:

nameserver <cluster‑dns‑ip>
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

CoreDNS runs as a set of containers (usually 2‑3 replicas) behind a kube‑dns Service. Its behavior is defined by a Corefile that chains plugins: cache, forward, loop, reload, loadbalance, health, prometheus, etc. Queries follow the plugin chain until a plugin returns a successful answer (e.g., from the cache or via forward to an upstream resolver) or returns REFUSED/SERVFAIL.

Understanding Search Paths and ndots Settings

The resolver library (glibc/musl) consults the search list when a query name contains fewer dots than the ndots threshold. For each entry in the search list, the resolver appends the suffix and issues a query. If none succeed, it finally tries the name as‑is (if it already had enough dots).

• ndots: Minimum number of dots in a name that makes the resolver treat it as “absolute” and skip the search list. Default in most Kubernetes pods is 5.
• search: Ordered list of domain suffixes appended to short names. The default list is derived from the pod’s namespace and the cluster domain.

Consequently, a name like redis (0 dots) triggers up to len(search)+1 queries when ndots > 0. A name like redis.backend (1 dot) still triggers searches if ndots > 1, etc.

How Search Paths Multiply Internal Lookups

Assume a pod in namespace web with the default search list:

search web.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

A request to redis proceeds as:

1. redis.web.svc.cluster.local
2. redis.svc.cluster.local
3. redis.cluster.local
4. redis (final attempt)

That is four UDP/TCP queries for a single application lookup. If the application also resolves redis:6379 (still a single name) the same multiplication occurs. With ndots:1, only names with zero dots would be expanded; a name like redis.backend (1 dot) would be sent as‑is, cutting the multiplication factor dramatically.

Role of ndots Settings in Controlling Lookup Traffic

Increasing ndots makes the resolver more aggressive in applying the search list, thereby increasing query volume. Decreasing ndots reduces the number of generated queries but may cause resolution failures for truly short names that rely on the search list (e.g., redis when you expect it to resolve to redis.default.svc.cluster.local). The trade‑off is between lookup safety and query load.

Example Use Cases: High ndots Values and Extended Search Paths

In a micro‑service heavy workload where each pod performs dozens of service lookups per second, moving from ndots:5 to ndots:1 can cut the DNS query rate by 60‑80 %, directly reducing pressure on CoreDNS.

CoreDNS Performance Under Load

CoreDNS Architecture and Configuration

CoreDNS is a plugin‑based DNS server written in Go. A typical Corefile for Kubernetes looks like:

.:53 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods verified
        upstream
        fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Key plugins affecting latency under high lookup volume:

• kubernetes: watches the API server for Services, Endpoints, Pods; builds an in‑memory map. Each query triggers a map lookup (O(1)) but requires a read lock on the underlying data structures.
• cache: stores recent NXDOMAIN and positive responses; reduces upstream forwarding.
• forward: sends unresolved queries to /etc/resolv.conf (often the node’s local resolver, e.g., systemd‑resolved). Each forward adds an RTT.
• prometheus: exports metrics (coredns_dns_response_size_bytes, coredns_dns_request_count_total, coredns_dns_response_ttl_bucket, etc.) useful for measuring load.

Measuring CoreDNS Performance with High Lookup Volumes

To generate controlled load, we can use dnsperf or k6 with a DNS plugin. Example using dnsperf:

# Install dnsperf (from https://github.com/DNS-OARC/dnsperf)
dnsperf -s <coredns-svc-ip> -d /tmp/queryfile -Q 50000 -T 60

-Q 50000 sends 50 k queries per second for 60 seconds. The query file can contain a mix of fully qualified names (to bypass search) and short names (to trigger search list expansion).

Metrics to watch (exposed on :9153/metrics):

CLI Examples: Monitoring CoreDNS Metrics and Logs

# 1. Port‑forward the metrics endpoint to localhost
kubectl -n kube-system port-forward svc/kube-dns 9153:9153 &
# 2. Scrape a snapshot
curl -s http://localhost:9153/metrics | grep coredns_dns_request_count_total
# 3. Enable query logging (add `log` plugin to Corefile, then rollout)
kubectl -n kube-system edit configmap coredns
#   Add:
#   log . {
#       class denial
#       name error
#   }
#   Then rollout:
kubectl rollout restart deployment/coredns -n kube-system
# 4. View logs for a pod
kubectl -n kube-system logs -l k8s-app=kube-dns -f | grep query

A sudden rise in coredns_dns_response_ttl_bucket{le="0"} together with flat or decreasing coredns_cache_hits_total indicates that many queries are falling through to the search list and returning NXDOMAIN (the resolver tried a suffix that does not exist).

Troubleshooting Latency Spikes in Kubernetes Applications

Identifying Latency Spikes with Kubernetes Tools

Application‑level latency can be measured with side‑car probes or using kubectl exec + time dig. Example:

# Measure DNS latency for a short name from inside a pod
kubectl exec -it <pod> -- sh -c "time dig +short redis @10.96.0.10"

Output includes real time (wall‑clock). Repeating this in a loop and collecting percentiles reveals spikes.

Kubernetes also provides DNS diagnostics via the kube-dns autoscaler metrics and the dnsPolicy field in pod specs. The kubectl top pod command shows CPU usage; if CoreDNS pods are consistently near their CPU limit, latency will rise.

Analyzing DNS Lookup Patterns and Search Path Configurations

You can inspect a pod’s effective resolv.conf:

kubectl exec <pod> -- cat /etc/resolv.conf

To see the search list and ndots value. Combine with get pod -o yaml to view the dnsConfig if overridden:

kubectl get pod <pod> -o yaml | grep -A5 dnsConfig

If you suspect excessive search‑list traffic, enable CoreDNS query logging (as above) and filter for queries that end in NXDOMAIN:

kubectl -n kube-system logs -l k8s-app=kube-dns | grep ";NXDOMAIN"

A high ratio of NXDOMAIN to NOERROR answers indicates wasted search‑list attempts.

Code Examples: Using Kubernetes APIs to Monitor DNS Performance

A small Go program that queries the Kubernetes API for all pods, extracts their dnsConfig, and aggregates the effective ndots value:

package main

import (
	"context"
	"fmt"
	"log"
	"strings"

	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
)

func main() {
	config, err := rest.InClusterConfig()
	if err != nil {
		log.Fatalf("failed to get in-cluster config: %v", err)
	}
	clientset, err := kubernetes.NewForConfig(config)
	if err != nil {
		log.Fatalf("failed to create clientset: %v", err)
	}

	pods, err := clientset.CoreV1().Pods("").List(context.TODO(), metav1.ListOptions{})
	if err != nil {
		log.Fatalf("failed to list pods: %v", err)
	}

	type ndotStat struct {
		total int
		count int
	}
	stats := make(map[string]ndotStat) // key = namespace

	for _, p := range pods.Items {
		ndots := 5 // default
		if p.Spec.DNSConfig != nil && p.Spec.DNSConfig.Options != nil {
			for _, opt := range p.Spec.DNSConfig.Options {
				if opt.Name == "ndots" {
					if _, err := fmt.Sscanf(opt.Value, "%d", &ndots); err == nil {
						// ndots parsed successfully
					}
				}
			}
		}
		key := p.Namespace
		s := stats[key]
		s.total += ndots
		s.count++
		stats[key] = s
	}

	for ns, s := range stats {
		avg := float64(s.total) / float64(s.count)
		fmt.Printf("Namespace %s: avg ndots = %.2f (based on %d pods)\n", ns, avg, s.count)
	}
}

Running this in a cluster gives you a quick view of whether namespaces are inadvertently using high ndots values (e.g., due to legacy dnsPolicy: Default).

Scaling Limitations of Kubernetes DNS Infrastructure

Horizontal Scaling of CoreDNS Deployments

CoreDNS scales horizontally by increasing the replica count of its Deployment. The kube‑dns Service uses ClusterIP with sessionAffinity: None, so client‑side load balancing (via kube‑proxy/IPVS) distributes UDP packets across replicas.

Limits:

• UDP connection tracking: Each replica maintains its own connection‑tracking state in the Linux conntrack table (if the node uses iptables). Very high QPS (> 150k pps per node) can exhaust conntrack buckets, causing packet drops.
• API server watch load: The kubernetes plugin watches Services, Endpoints, Pods. Each watch consumes a stream; many replicas multiply the watch load on the API server. Beyond ~30‑40 replicas, API server CPU can become the bottleneck.
• Lock contention: The internal Service/Endpoint map is protected by a sync.RWMutex. Under heavy read‑only workload (most DNS queries are reads), contention is low, but bursts of updates (e.g., scaling a large Deployment) can cause short stalls.

Best practice: Start with 2‑3 replicas per 1000 pods, monitor coredns_dns_request_count_total per replica, and add replicas when any instance exceeds ~70 % CPU or when the 99th‑percentile latency exceeds your SLA (e.g., 5 ms).

Vertical Scaling of CoreDNS Instances

Increasing CPU and memory limits allows each CoreDNS process to handle more concurrent queries and a larger cache.

• CPU: CoreDNS is single‑threaded per plugin chain but uses Go’s runtime scheduler; allocating 2 vCPU per replica typically yields ~50k QPS on a modern Xeon with UDP. Adjust based on observed coredns_dns_request_count_total and latency metrics.