Migrating from kube dns to CoreDNS without brownouts

Introduction to Kube-DNS and CoreDNS

Overview of Kube-DNS

Kube‑DNS is the legacy DNS add‑on for Kubernetes clusters, composed of three containers: kubedns (SkyDNS‑based server), dnsmasq (caching and stub‑domain forwarding), and sidecar (health‑checking and metrics). It runs as a Deployment with a ClusterIP service named kube-dns in the kube-system namespace. Kube‑DNS watches the Kubernetes API for Services, Endpoints, and Pods, synthesizing DNS records in the cluster.local zone. Its plugin architecture is limited to compiled‑in SkyDNS logic; extending behavior requires rebuilding the binary or adding side‑car containers.

Overview of CoreDNS

CoreDNS is a modular, Go‑based DNS server that has become the default DNS provider for Kubernetes since v1.13. It runs as a single container (or multiple replicas) and uses a plugin chain defined in a Corefile. CoreDNS ships with plugins for Kubernetes service discovery (kubernetes), caching (cache), load balancing (loadbalance), health checks (health), metrics (prometheus), logging (log), and many others. Because the plugin chain is configurable at runtime, operators can add, remove, or reorder functionality without rebuilding the binary. CoreDNS exposes metrics via /metrics and can emit structured logs to stdout.

Motivation for Migration

Migrating from Kube‑DNS to CoreDNS addresses several operational risks:

• Extensibility: CoreDNS plugins let operators add custom rewrite, federation, or security policies without recompiling.
• Observability: Built‑in Prometheus metrics and structured logging reduce reliance on side‑car exporters.
• Resource efficiency: A single CoreDNS container replaces the three‑container Kube‑DNS pod, lowering memory and CPU overhead.
• Community support: CoreDNS receives active upstream development; Kube‑DNS is effectively in maintenance mode.
• Feature parity: CoreDNS implements the same Kubernetes service discovery semantics while offering additional capabilities like autopath, proxy, and reload.

Execution risk stems from changing a critical control‑plane service. A staged migration with explicit proof points limits blast radius and provides rollback windows before any cluster‑wide cutover.

Pre-Migration Planning and Preparation

Cluster Requirements and Assessment

1. Kubernetes version: CoreDNS is supported on Kubernetes ≥1.11; verify control plane and kubelet versions meet the minimum.
2. Resource headroom: Measure current Kube‑DNS CPU/Memory usage (kubectl top pod -n kube-system). Ensure nodes have sufficient spare capacity for at least two CoreDNS replicas (typically 50 mCPU and 64 MiB each) plus a buffer for canary traffic.
3. Network policy: Confirm no NetworkPolicy blocks UDP/TCP port 53 from pods to the kube-dns Service IP. CoreDNS will listen on the same port.
4. Custom Kube‑DNS configuration: Export the existing ConfigMap (kube-dns) and any associated kube-dns-autoscaler ConfigMap. Note custom stubDomains, upstream nameservers, or nodeLocalDNSCache settings that must be reproduced in CoreDNS.
5. Version pinning: Choose a CoreDNS image tag (e.g., v1.10.1) and record it in a version‑controlled manifest to avoid drift.

CoreDNS Configuration and Customization

Create a base Corefile that mirrors Kube‑DNS behavior:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        log
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods verified
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

• log enables structured request/response logging.
• health provides liveness/readiness endpoints used by Kubernetes probes.
• kubernetes plugin replicates service discovery; pods verified ensures only pod A records for existing pods are returned.
• forward handles external namespaces (non‑cluster.local) using the node’s /etc/resolv.conf.
• cache 30 sets a 30‑second TTL for cached entries, similar to Kube‑DNS/dnsmasq.
• reload allows automatic Corefile reloads when the ConfigMap changes.
• loadbalance distributes queries among multiple upstream endpoints (useful when using forward).

Customizations (stubDomains, upstream servers, autopath) are added as additional plugin blocks or parameters to the kubernetes block.

Testing Strategy and Tools

• Unit validation: Use coredns -conf ./Corefile -dns.port 10053 locally to verify syntax (coredns -conf ./Corefile -validate).
• Functional tests: Deploy a test pod (dnstools) that runs dig @<svc-ip> <name> +tcp +tries=1 +time=2 against both Kube‑DNS and CoreDNS.
• Latency measurement: Use dnsviz or bash loops with time dig to capture query latency distributions.
• Cache hit ratio: Enable CoreDNS cache plugin metrics (coredns_cache_hits_total, coredns_cache_misses_total) and compare with Kube‑DNS dnsmasq_cache_inserts_total/dnsmasq_cache_hits_total (if exported).
• Canary routing: Leverage Kubernetes EndpointSlice or Service annotations (trafficSplit.kubernetes.io) or a service mesh (Istio/Linkerd) to shift a percentage of UDP/TCP 53 traffic to CoreDNS.
• Automated verification: Write a Helm test or a Kubernetes Job that runs a battery of DNS queries against both services and asserts parity (response codes, answer sections, latency p95 < X ms).

Staged Migration Approach

Phase 1: Parallel Deployment

Deploying CoreDNS in Parallel with Kube-DNS

1. Create the CoreDNS Deployment (replicas = 2 for HA):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "CoreDNS"
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: kube-dns
  template:
    metadata:
      labels:
        k8s-app: kube-dns
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: kube-dns
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      nodeSelector:
        kubernetes.io/os: linux
      containers:
      - name: coredns
        image: k8s.gcr.io/coredns:v1.10.1
        args: ["-conf", "/etc/coredns/Corefile"]
        volumeMounts:
        - name: config-volume
          mountPath: /etc/coredns
          readOnly: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          timeoutSeconds: 5
          periodSeconds: 10
          failureThreshold: 5
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
          periodSeconds: 10
          failureThreshold: 3
      volumes:
      - name: config-volume
        configMap:
          name: coredns
          items:
          - key: Corefile
            path: Corefile
      dnsPolicy: Default  # Do not use Kubernetes DNS for the coredns pod itself

2. Create a matching Service (same ClusterIP as Kube‑DNS to avoid client reconfiguration):

apiVersion: v1
kind: Service
metadata:
  name: kube-dns
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/name: "CoreDNS"
spec:
  selector:
    k8s-app: kube-dns
  clusterIP: <existing-kube-dns-clusterIP>  # Preserve the original IP
  ports:
  - name: dns
    port: 53
    protocol: UDP
  - name: dns-tcp
    port: 53
    protocol: TCP
  - name: metrics
    port: 9153
    protocol: TCP

Note: The Service’s clusterIP must be copied from the existing Kube‑DNS Service (kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}'). This ensures that pods continue to resolve via the same IP without needing a rolling update of /etc/resolv.conf.

3. Verify coexistence:
   • kubectl get endpoints kube-dns -n kube-system should show both Kube‑DNS and CoreDNS endpoints.
   • DNS queries from a test pod should be answered by either backend (load‑balancing is performed by kube-proxy).

Configuring CoreDNS for Coexistence with Kube-DNS

• Ensure the CoreDNS kubernetes plugin does not conflict with Kube‑DNS’s internal service discovery. Both plugins will serve the same zone; the first to respond wins. To avoid duplicate answers, enable the loop plugin (already in the Corefile) which detects and drops queries that would cause infinite loops.
• If you prefer to keep Kube‑DNS as the authoritative source and use CoreDNS only for caching/federation, adjust the kubernetes block to fallthrough and place a proxy plugin upstream to the Kube‑DNS Service IP. For a pure parallel test, the simple configuration above suffices.

Phase 2: Canary Testing

Selective Traffic Routing to CoreDNS

Because the Service IP is shared, traffic splitting must happen at the proxy level. Two common approaches:

1. kube-proxy IPVS mode with trafficSplit annotation (requires a custom kube-proxy build or a third‑party controller).
2. Service Mesh (Istio/Linkerd) – define a DestinationRule that splits 5 % of UDP/TCP 53 traffic to the CoreDNS subset.

Example using Istio (assuming Istio is installed and the kube-dns Service is added to the mesh):

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: kube-dns
  namespace: kube-system
spec:
  host: kube-dns.kube-system.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
  subsets:
  - name: kube-dns
    labels:
      k8s-app: kube-dns
      kubernetes.io/name: kube-dns   # Kube‑DNS pod label
  - name: coredns
    labels:
      k8s-app: kube-dns
      kubernetes.io/name: CoreDNS    # CoreDNS pod label
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kube-dns
  namespace: kube-system
spec:
  hosts:
  - kube-dns.kube-system.svc.cluster.local
  http:
  - route:
    - destination:
        host: kube-dns.kube-system.svc.cluster.local
        subset: kube-dns
      weight: 95
    - destination:
        host: kube-dns.kube-system.svc.cluster.local
        subset: coredns
      weight: 5

Important: Istio’s UDP support is limited; for pure UDP DNS you may need to enable Sidecar with outboundTrafficPolicy.mode: REGISTRY_ONLY and rely on the mesh’s TCP fallback, or use a Layer‑4 load‑balancer (e.g., MetalLB) with weighted services.

If a service mesh is unavailable, use a NodePort or LoadBalancer service for CoreDNS only and update a subset of pods’ /etc/resolv.conf via a DaemonSet that rewrites the file (this is more invasive but works for canary validation).

Monitoring and Comparison of Kube-DNS and CoreDNS

Deploy a monitoring Job that runs every minute:

apiVersion: batch/v1
kind: Job
metadata:
  name: dns-canary-check
  namespace: kube-system
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: check
        image: appropriate/curl:latest
        command:
        - /bin/sh
        - -c
        - |
          NAMES="kubernetes.default.svc.cluster.local google.com internal.myapp.svc.cluster.local"
          for name in $NAMES; do
            for proto in udp tcp; do
              # Query Kube‑DNS via the service IP (still the same)
              KUBE_ANS=$(dig @<kube-dns-clusterIP> $name +$proto +tries=1 +time=2 +short)
              # Query CoreDNS directly via its pod IP (obtained via downward API)
              CORE_ANS=$(dig @$MY_POD_IP $name +$proto +tries=1 +time=2 +short)
              if [ "$KUBE_ANS" != "$CORE_ANS" ]; then
                echo "Mismatch for $name ($proto): Kube=$KUBE_ANS Core=$CORE_ANS"
                exit 1
              fi
            done
          done
          echo "All queries matched"
        env:
        - name: MY_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP

• Metrics to watch (Prometheus):
  • coredns_cache_hits_total vs dnsmasq_cache_hits_total (if exported).
  • coredns_request_duration_seconds_bucket (latency histogram).
  • coredns_response_rcode_total (ensure NOERROR dominates).
  • kube_dns_latency_seconds (if you have an exporter for Kube‑DNS; otherwise rely on the Job’s latency measurements).
• Success criteria for canary:
  • 99.9 % of queries return identical answer sections.
  • Latency p95 increase < 2 ms compared to baseline.
  • Cache hit ratio within ±5 % of Kube‑DNS baseline.
  • No increase in coredns_response_rcode_total for SERVFAIL or REFUSED.

If any criterion fails, the canary is halted and traffic is shifted back to 100 % Kube‑DNS (by adjusting the VirtualService weight or removing the CoreDNS endpoints from the Service).

Phase 3: Cluster-Wide Cutover

Switching to CoreDNS as the Primary DNS Service

1. Update the Service to remove Kube‑DNS pods from the selector (or scale Kube‑DNS Deployment to 0). Keeping the same