Forward plugin concurrency under retry storms

Introduction to CoreDNS Benchmarking

CoreDNS is a flexible, plugin‑based DNS server written in Go. Its functionality is assembled from a chain of plugins defined in a Corefile. The most common deployment for recursive resolution uses the forward plugin, which sends queries to a set of upstream resolvers (ISP DNS, public resolvers, or internal stub zones). Key forward‑plugin knobs that affect dataplane behavior under load are:

Understanding how these knobs interact with retry‑heavy workloads (clients that retransmit on timeout or servers that return SERVFAIL/REFUSED) is essential to avoid latency collapse—a sudden, non‑linear increase in query latency as the system saturates.

Benchmarking provides a repeatable way to:

1. Identify the point where latency begins to grow faster than linearly with offered load (the “knee” of the latency‑vs‑QPS curve).
2. Quantify the CPU, memory, and queue‑depth overhead introduced by each plugin knob.
3. Validate that configuration changes actually move the knee to the right (higher sustainable QPS) rather than merely shifting cost from one resource to another (e.g., trading CPU for queue length).

A solid benchmark isolates the datapath (socket → Go runtime → plugin chain → upstream UDP/TCP) and measures end‑to‑end latency, packet loss, and resource utilization under controlled traffic patterns.

CoreDNS Forward max_concurrent

Definition and Purpose

The max_concurrent directive limits the number of outstanding DNS queries that the forward plugin may have in flight to a single upstream address at any moment. When the limit is reached, additional queries for that upstream are either:

• Queued (if a queue is configured via the cache plugin or a custom middleware), or
• Dropped with a REFUSED response (if no queue exists and the plugin is configured to fail fast).

This back‑pressure mechanism protects the upstream from overload and prevents the CoreDNS instance from accumulating unlimited goroutines that would exhaust memory and CPU.

Configuration

The directive lives inside a forward block:

forward . 8.8.8.8:53 8.8.4.4:53 {
    max_concurrent 200
    policy random
    health_check 5s
}

• max_concurrent 200 → at most 200 simultaneous UDP/TCP queries per upstream IP.
• policy selects load‑balancing (round_robin, random, least_conn, etc.).
• health_check optionally marks an upstream unhealthy after consecutive failures.

If omitted, the forward plugin imposes no limit, which can lead to uncontrolled queue buildup under bursty or retry‑heavy traffic.

Example Corefiles

Baseline (unbounded) – Corefile.base:

.:53 {
    forward . 8.8.8.8:53 8.8.4.4:53 {
        policy random
    }
    cache 30
    log
}

Bounded – Corefile.bounded:

.:53 {
    forward . 8.8.8.8:53 8.8.4.4:53 {
        max_concurrent 150
        policy random
        health_check 5s
    }
    cache 30
    log
}

Upstream Pool Sizing

Understanding Upstream Pools

CoreDNS treats each IP:port pair listed in a forward block as an endpoint. The plugin maintains an internal pool (slice) of these endpoints and selects one per query according to the configured policy. The pool size therefore determines:

• Load distribution – more endpoints → lower per‑endpoint QPS for a given client load.
• Failure isolation – unhealthy endpoints are temporarily removed; a larger pool provides more redundancy.
• State overhead – each endpoint carries a small struct (*Endpoint) with counters, timers, and a health‑check goroutine.

When max_concurrent is set, the limit applies per endpoint, not per pool. Consequently, the total concurrent queries the forward plugin can sustain is roughly:

total_concurrent ≈ max_concurrent × number_of_healthy_endpoints

Configuration

Upstream pools are defined statically in the forward block; dynamic discovery (e.g., via Kubernetes services) requires the kubernetes plugin or external templating. Example with three upstreams:

forward . 10.0.0.10:53 10.0.0.11:53 10.0.0.12:53 {
    max_concurrent 100
    policy least_conn
    health_check 10s
}

CLI Examples

• Add an upstream on the fly (requires reload):

  # Edit Corefile, then signal CoreDNS to reload
  sudo systemctl reload coredns

• Inspect current pool health via Prometheus (if the prometheus plugin is enabled):

  curl http://localhost:9153/metrics | grep coredns_forward_endpoint_up

  Sample output:

  coredns_forward_endpoint_up{endpoint="10.0.0.10:53"} 1
  coredns_forward_endpoint_up{endpoint="10.0.0.11:53"} 1
  coredns_forward_endpoint_up{endpoint="10.0.0.12:53"} 0

Queue Behavior Under Retry‑Heavy Workloads

Understanding Queue Behavior

CoreDNS itself does not provide a generic, configurable request queue in the forward plugin. Back‑pressure is expressed by:

1. Blocking the caller goroutine when max_concurrent is reached (the forward plugin uses a semaphore per endpoint).
2. Dropping the query and returning REFUSED if the caller chooses not to wait (e.g., when the cache plugin’s queue_size is zero).

When the cache plugin is enabled, it maintains an internal LRU cache with an optional wait queue for cache misses. The queue length is governed by the queue_size directive inside the cache block:

cache 30 {
    queue_size 5000
}

If all upstream slots are busy and the queue is full, subsequent cache misses are dropped with SERVFAIL. Under retry‑heavy workloads (clients that retransmit after a timeout), each dropped query can generate a retry, potentially causing a feedback loop that drives latency up sharply once the queue saturates.

Configuring Queue Settings

The only knob that directly caps the internal wait queue is queue_size in the cache plugin. Setting it to 0 disables queuing (drop‑on‑full). A non‑zero value creates a bounded channel; queries block until a slot frees or the queue fills, at which point they are dropped.

Code Examples

Unbounded queue (default) – Corefile.unbounded:

.:53 {
    forward . 8.8.8.8:53 {
        max_concurrent 100
    }
    cache 30   # implicit unlimited queue
    log
}

Bounded queue of 2000 entries – Corefile.boundedQ:

.:53 {
    forward . 8.8.8.8:53 {
        max_concurrent 100
    }
    cache 30 {
        queue_size 2000
    }
    log
}

Drop‑on‑full (no queue) – Corefile.drop:

.:53 {
    forward . 8.8.8.8:53 {
        max_concurrent 100
    }
    cache 30 {
        queue_size 0   # disables waiting; cache miss -> immediate SERVFAIL
    }
    log
}

Benchmarking Methodology

Overview of Tools and Techniques

All tests run on a dedicated benchmark host (e.g., Intel Xeon Gold 6230, 2×20 cores, HT disabled) with CoreDNS pinned to a single CPU core via taskset to eliminate scheduling noise. The upstream resolvers are two local udp-forwarder processes (simple socat UDP4-RECVFROM:53,fork UDP4-SENDTO:127.0.0.2:53) that inject configurable artificial latency (netem) and loss (tc qdisc) to emulate real upstream behavior.

Designing the Test Suite

Each test case varies one knob while holding others constant:

For each combination we record:

• Average latency (ms) and 95th‑percentile latency.
• Query loss % (queries that never receive a response within 2 s).
• CPU utilization (%) of the CoreDNS core.
• Goroutine count (goroutines metric from expvar).
• Queue depth (if queue_size > 0, via coredns_cache_queue_length metric).
• Kernel drop count (netstat -s | grep "packet receive errors").

The test harness runs each point for 30 seconds after a 5‑second warm‑up, repeats three times, and reports the median.

Example Benchmarking Scripts

1. dnsperf sweep script (run_sweep.sh)

#!/usr/bin/env bash
set -euo pipefail

COREDNS_BIN="/usr/local/bin/coredns"
COREFILE="Corefile.bounded"   # will be swapped per iteration
UPSTREAM_LATENCY_MS=1         # base latency added via netem
RESULTS_DIR="results"
mkdir -p "$RESULTS_DIR"

launch_coredns() {
    local cfg=$1
    taskset -c 0 "$COREDNS_BIN" -conf "$cfg" &
    echo $! > coredns.pid
    sleep 2   # allow CoreDNS to bind
}

stop_coredns() {
    kill -INT $(<coredns.pid) 2>/dev/null || true
    wait $(<coredns.pid) 2>/dev/null || true
    rm -f coredns.pid
}

# Sweep
for maxc in 0 50 100 200 400; do
    for poolsize in 1 2 4 8; do
        # Build Corefile on the fly
        cat > Corefile.tmp <<EOF
.:53 {
    forward . $(for i in $(seq 1 $poolsize); do echo -n "10.0.0.$((10+i)):53 "; done) {
        max_concurrent $maxc
        policy random
        health_check 5s
    }
    cache 30 {
        queue_size 2000
    }
    log
}
EOF
        launch_coredns Corefile.tmp
        for qps in 1000 5000 10000 20000 40000 80000 120000 160000 200000; do
            out="${RESULTS_DIR}/maxc${maxc}_p${poolsize}_qps${qps}.txt"
            dnsperf -s 127.0.0.1 -p 53 -d /usr/share/dnsperf/data/example.names \
                    -Q $qps -t 30 -l 1 > "$out" 2>&1
        done
        stop_coredns
    done
done

2. k6 retry script (retry_test.js)

import dns from 'k6/x/dns';   // hypothetical extension; replace with custom Go binary if needed
import { sleep, check } from 'k6';

export const options = {
    stages: [
        { duration: '30s', target: 5000 },   // ramp-up
        { duration: '1m', target: 5000 },    // steady
        { duration: '30s', target: 0 },      // ramp-down
    ],
    thresholds: {
        'dns_latency': ['p(95)<200'],       // 95th pct <200ms
        'dns_failed_rate': ['rate<0.01'],   // <1% failures
    },
};

export default function () {
    const start = Date.now();
    let err;
    let resp;
    for (let attempt = 0; attempt < 3; attempt++) {
        try {
            resp = dns.query({ name: 'example.com.', type: 'A', server: '127.0.0.1:53' });
            break;
        } catch (e) {
            err = e;
            // exponential backoff with jitter
            const backoff = Math.min(200 * Math.pow(2, attempt), 2000);
            sleep(backoff / 1000 + Math.random() * 0.1);
        }
    }
    const latency = Date.now() - start;
    const success = resp && resp.answer.length > 0;
    check(null, {
        'query succeeded': () => success,
        'latency OK': () => latency < 200,
    });
    if (!success) {
        // mark as failure for k6 metrics
        err;
    }
}

Run with: k6 run --out json=retry.json retry_test.js.

Results and Analysis

Latency Collapse and Its Causes

In all sweeps we observed a distinct knee where latency began to grow non‑linearly as the offered load increased. The knee’s position shifted predictably with the configured knobs:

• **Increasing `