Neighbor cache GC under 100k endpoint churn

Introduction to Benchmarking Linux ARP and NDP

Address Resolution Protocol (ARP) maps IPv4 addresses to link‑layer (MAC) addresses on Ethernet networks. Neighbor Discovery Protocol (NDP) performs the same function for IPv6, using ICMPv6 messages (Neighbor Solicitation and Neighbor Advertisement) and maintaining a neighbor cache. Both protocols rely on a kernel‑maintained neighbour table to avoid repeated resolution for each packet. When entries become stale, the kernel runs a garbage collector (GC) to reclaim memory and keep the cache size bounded.

In environments with dense, short‑lived endpoints (container workloads, VM live‑migration, test harnesses that repeatedly create and destroy IP addresses), the neighbour table experiences high churn. If GC thresholds are too low, the kernel invokes GC too frequently, consuming CPU cycles and adding lookup latency. If thresholds are too high, the cache can grow unchecked, leading to memory pressure, longer linear search times, and eventual packet loss when the table overflows. Benchmarking GC behavior lets operators locate the point where CPU pressure, lookup latency, and packet loss begin—often before switch‑level counters (e.g., port discards) show any anomaly.

Understanding Neighbor Garbage Collection Thresholds

ARP Cache Garbage Collection

The ARP neighbour table is governed by three thresholds in /proc/sys/net/ipv4/neigh/<dev>/gc_thresh* (or the default subtree for global defaults):

• gc_thresh1 – soft limit; entries above this are eligible for lazy reclamation.
• gc_thresh2 – hard limit; when exceeded, the kernel forces immediate GC.
• gc_thresh3 – maximum absolute size; further allocations fail with ENOBUFS.

Additional parameters influence GC timing:

• gc_interval – milliseconds between automatic GC runs (default 30 000 ms).
• gc_stale_time – seconds after which an entry is considered stale (default 60 s).
• unres_qlen – maximum number of pending packets while waiting for a resolution (default 3).

When the number of entries exceeds gc_thresh2, the kernel walks the hash buckets, removes entries that are INCOMPLETE or STALE and have not been used recently, and frees the associated struct neighbour. The walk is O(N) in the number of buckets, which can cause noticeable CPU spikes if the table is large.

NDP Cache Garbage Collection

IPv6 uses the same neighbour subsystem; the relevant sysctls live under /proc/sys/net/ipv6/neigh/<dev>/gc_thresh*. The semantics are identical to ARP, but NDP adds:

• retrans_time – milliseconds between retransmissions of Neighbor Solicitation (default 1000 ms).
• base_reachable_time – used to compute reachable time (default 30 000 ms).

Because IPv6 addresses are 128‑bit, hash collisions are rarer, but the same GC algorithm applies. In kernels ≥ 5.4, the neighbour table uses a seqlock‑protected hash table with per‑bucket locks, reducing contention but not eliminating the O(N) scan during forced GC.

Impact of Ephemeral Endpoint Churn on Garbage Collection

Ephemeral endpoint churn creates a pattern:

1. Rapid allocation of new neighbour entries (often INCOMPLETE until resolution succeeds).
2. Quick transition to STALE after the first use, then to DELAY/PROBE as the entry ages.
3. Frequent deletions when the endpoint disappears (e.g., container pod deletion).

If the churn rate exceeds the rate at which GC can reclaim memory, the table will hover near gc_thresh2 or gc_thresh3. Each forced GC scan consumes CPU cycles proportional to the table size, increasing per‑packet lookup latency (the neighbour lookup must traverse the hash bucket and may need to wait for GC to finish). When the table hits gc_thresh3, new allocations drop packets with ENOBUFS, which appears as TX drops at the socket layer before any switch sees a loss.

Setting Up the Benchmarking Environment

Configuring Linux Networking Parameters

Start with a clean baseline (kernel 5.15 or newer recommended for improved neighbour scalability). On a test host with at least two physical or virtual NICs (e.g., eth0 for traffic, eth1 for management), apply the following sysctls to observe GC behaviour:

# Show current defaults
sysctl -a | grep neigh.default.gc_thresh

# Set low thresholds to trigger GC early (for demonstration)
sysctl -w net.ipv4.neigh.default.gc_thresh1=128
sysctl -w net.ipv4.neigh.default.gc_thresh2=256
sysctl -w net.ipv4.neigh.default.gc_thresh3=512

# IPv6 equivalents
sysctl -w net.ipv6.neigh.default.gc_thresh1=128
sysctl -w net.ipv6.neigh.default.gc_thresh2=256
sysctl -w net.ipv6.neigh.default.gc_thresh3=512

# Reduce GC interval to see more frequent runs
sysctl -w net.ipv4.neigh.default.gc_interval=5000   # 5 s
sysctl -w net.ipv6.neigh.default.gc_interval=5000

Persist changes via /etc/sysctl.d/99-neighbour-gc.conf if needed.

Installing Required Tools and Software

Creating a Test Scenario for Ephemeral Endpoint Churn

We simulate churn using network namespaces and veth pairs. Each iteration creates a unique IPv4/IPv6 address, brings the interface up, sends a single ping (to trigger ARP/NDP resolution), then tears down the namespace. The script below runs at a configurable rate (e.g., 5 k endpoints / s).

#!/usr/bin/env bash
# churn.sh – generate ephemeral endpoints at rate $RATE per second
RATE=${1:-5000}          # endpoints per second
DURATION=${2:-30}        # seconds to run
IFACE=eth0               # NIC attached to the test network
BASE_IPV4=10.0.0
BASE_IPV6=2001:db8:1::

end=$((SECONDS+DURATION))
while [ $SECONDS -lt $end ]; do
    # Create a unique namespace
    NS="ns-$(printf '%08x' $RANDOM)"
    ip netns add $NS

    # Create veth pair
    VPEER="vpe-${NS}"
    VLOC="vl-${NS}"
    ip link add $VLOC type veth peer name $VPEER
    ip link set $VLOC netns $NS
    ip link set $VPEER up

    # Assign addresses in the namespace
    ip netns exec $NS ip addr add ${BASE_IPV4}.$((RANDOM%254+2))/24 dev $VLOC
    ip netns exec $NS ip addr add ${BASE_IPV6}::$(printf '%04x' $RANDOM)/64 dev $VLOC
    ip netns exec $NS ip link set $VLOC up
    ip netns exec $NS ip link set lo up

    # Trigger ARP/NDP resolution by pinging the gateway (assume .1/::1)
    ip netns exec $NS ping -c 1 -W 1 ${BASE_IPV4}.1 >/dev/null 2>&1
    ip netns exec $NS ping6 -c 1 -W 1 ${BASE_IPV6}::1 >/dev/null 2>&1

    # Cleanup
    ip link set $VPEER down
    ip link del $VPEER
    ip netns delete $NS

    # Sleep to maintain rate
    sleep=$(awk "BEGIN {print 1.0/$RATE}")
    # Use nanosleep via bash builtin for sub‑second precision
    # (busywait for sub‑ms rates)
    if (( $(echo "$sleep < 0.001" | bc -l) )); then
        continue
    else
        sleep $sleep
    fi
done

Make the script executable (chmod +x churn.sh) and run it on the test host while monitoring the neighbour table and CPU.

Benchmarking Methodology

Measuring CPU Pressure and Lookup Latency

• CPU pressure – Use pidstat -p ALL 1 to capture per‑process CPU usage, focusing on ksoftirqd and kworker threads that handle neighbour GC. Additionally, sample perf stat -e cycles,instructions,cache-references,cache-misses -a sleep 30 to get overall IPC and stall cycles.

• Lookup latency – Trace the neigh_lookup entry point with bpftrace:

  bpftrace -e '
  tracepoint:net:neigh_lookup {
      @lat[comm] = hist((nsecs - arg0)/1000); // convert to µs
  }
  interval:s:1 {
      print(@lat);
      clear(@lat);
  }
  '

  The histogram shows the distribution of neighbour lookup latency (time spent in the kernel before returning the MAC address). An increase in the median or 99th‑percentile indicates GC contention.

• GC activity – Export neighbour statistics via /proc/net/stat/arp_cache and /proc/net/stat/ndisc_cache. Fields such as gc_runs, gc_collected, and gc_failed give a direct view of how often the kernel is forced to collect.

Monitoring Packet Loss and Switch Counters

• Host‑side loss – Observe tx_drops and rx_drops from ethtool -S eth0 and ip -s link show eth0. Increments in tx_drops often correspond to ENOBUFS when the neighbour table is full.
• Switch counters – If the test NIC is connected to a managed switch, poll SNMP OIDs ifOutDiscards and ifInDiscards (or use the switch CLI) every second. The goal is to see whether host‑side loss appears before any rise in switch discards.
• Packet capture – Run tcpdump -i eth0 -nn -e -c 10000 arp or ip6 to verify that ARP/NDP requests are being generated and that replies are not being dropped due to lack of neighbour entries.

Using CLI Tools for Benchmarking

• Baseline measurement – Before starting churn, capture neighbour table size:

  ip -4 neigh show | wc -l   # IPv4 entries
  ip -6 neigh show | wc -l   # IPv6 entries

• During churn – Sample every second:

  watch -n 1 "ip -s neigh show eth0 | grep -E 'reachable|stalled|failed'"

• After churn – Verify that the table has returned to near‑zero:

  ip -4 neigh flush all
  ip -6 neigh flush all

Troubleshooting Common Issues

Identifying and Resolving ARP Cache Exhaustion

Symptoms: Rising tx_drops on the NIC, dst cache full messages in dmesg, and neigh_lookup latency histograms shifting right.

Diagnostic steps:

1. Check net.ipv4.neigh.default.gc_thresh3 – if the current neighbour count approaches this value, the kernel will start dropping packets.
2. Examine /proc/net/stat/arp_cache for gc_failed increments.
3. Increase gc_thresh3 (and optionally gc_thresh2) in steps of 2×, monitoring CPU usage to ensure GC does not become a constant background load.
4. If the NIC has hardware offload (e.g., ARP offload), verify that it is enabled (ethtool -k eth0 | grep arp-offload). Offloading can reduce host‑side processing but may mask GC pressure; disable it for a pure software benchmark if needed.

Return only the finalized markdown text as requested.