Exporter overhead on already busy routers

Introduction

High‑frequency scraping of network interface statistics and TX/RX queues forces the kernel to copy counters from per‑CPU or per‑queue structures into user space on each poll. Each poll incurs CPU cycles for system‑call entry/exit, copying of 64‑bit counters, and any required locking (e.g., u64_stats_sync). It also generates memory traffic as counters are read from cache lines that may be dirty due to NAPI processing, and can cause contention if the scraper runs on the same core that handles NAPI poll loops. The tax scales roughly linearly with poll frequency until the overhead saturates a core or exhausts memory bandwidth, at which point packet forwarding begins to drop or latency spikes.

Benchmarking provides a repeatable method to isolate the cost of the scraping loop from the cost of actual traffic, yields metrics such as CPU % per kHz of poll and extra memory bandwidth per poll, and helps decide when to switch from polling to event‑driven mechanisms (netlink notifications, eBPF maps, XDP‑driven counters).

Setting Up the Benchmarking Environment

Choosing the Right Tools and Software

Configuring Linux and Routing Hosts for Benchmarking

1. Disable frequency scaling

   echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

2. Isolate cores for the scraper (and optionally the traffic generator) using isolcpus and taskset. Example kernel boot line:

   isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3

   Bind the scraper to CPU 2:

   taskset -c 2 ./scraper -i eth0 -i 1000   # 1 kHz poll

3. Turn off unnecessary offloads

   ethtool -K eth0 tso off gso off gro off lro off

4. Verify NAPI weighting (default usually 64)

   ethtool -c eth0 | grep -i napi

5. Set static IRQ affinity to keep NIC queues on dedicated cores (e.g., cores 4‑7)

   echo 4-7 > /proc/irq/<irq-number>/smp_affinity_list

6. Mount debugfs for eBPF/bpftrace if needed

   mount -t debugfs none /sys/kernel/debug

Installing and Configuring Necessary Packages

On Ubuntu 22.04 (or equivalent):

apt-get update
apt-get install -y build-essential linux-tools-common linux-tools-`uname -r` \
                   ethtool iproute2 pktgen-dkms trex \
                   perf bpftrace python3-psutil prometheus-node-exporter grafana

Compile the scraper (see Code Examples) with -O2 -march=native -flto.

Benchmarking CPU Tax

Using System Monitoring Tools to Track CPU Usage

• Per‑process CPU: pidstat -p <scraper-pid> 1 reports %CPU and voluntary/involuntary context switches.
• System‑wide: perf stat -e cycles,instructions,cache-references,cache-misses -p <scraper-pid> sleep 10 gives cycles per second.
• Core‑level: mpstat -P ALL 1 shows per‑core utilization; watch the isolated core’s %user rise with poll rate.

Running Benchmarking Tests with High‑Frequency Interface and Queue Scraping

Scraper loop (simplified C):

while (!stop) {
    clock_gettime(CLOCK_MONOTONIC, &start);
    /* read rx/tx packets, bytes, drops, errors from sysfs */
    read_counter("/sys/class/net/eth0/statistics/rx_packets", &rx_pkt);
    read_counter("/sys/class/net/eth0/statistics/tx_packets", &tx_pkt);
    /* optional: per‑queue stats */
    for (int q = 0; q < num_queues; ++q) {
        char path[128];
        snprintf(path, sizeof(path),
                 "/sys/class/net/eth0/queues/rx-%d/packets", q);
        read_counter(path, &rx_q[q]);
    }
    clock_gettime(CLOCK_MONOTONIC, &end);
    /* sleep remaining time to hit target frequency */
    nanosleep(&rem, NULL);
}

Compile and run with varying -i (interval in µs):

# Baseline: 10 ms (100 Hz)
taskset -c 2 ./scraper -i 10000 -i eth0 &
# Sweep
for us in 1000 500 200 100 50 20 10; do
    taskset -c 2 ./scraper -i $us -i eth0 &
    scraper_pid=$!
    sleep 5
    pidstat -p $scraper_pid 1 3
    kill $scraper_pid
done

While the scraper runs, the traffic generator pushes a constant 5 Gbps UDP stream (≈7.5 Mpps 64‑byte packets) to stress the NIC and NAPI.

Analyzing CPU Usage Patterns and Identifying Bottlenecks

• Example observations on Xeon E5‑2680 v4, Linux 5.15:

  • 100 Hz → scraper %user ≈ 0.8 % on core 2, negligible impact on NAPI cores.
  • 1 kHz → %user ≈ 7.5 %, context switches ↑ from 120/s to 1.2 k/s.
  • 10 kHz → %user ≈ 68 %, scraper saturates core 2; NAPI cores see a 3‑5 % rise in %softirq due to increased lock contention on u64_stats_sync.
  • 20 kHz → %user > 90 %, packet loss appears on the receiver (RX drops ↑ from 0 to ~0.02 % of packets) as the scraper steals cycles from the softirq handling NAPI poll loop.

• Bottleneck identification: perf top shows the hottest symbol as __u64_stats_sync (spinlock protecting per‑CPU u64_stats) followed by vfs_read (sysfs attribute reads). Lock contention grows with poll frequency because each read acquires the same seqlock for each counter.

• Memory tax: negligible (< 0.1 % of memory bandwidth) because each read is 8 bytes; the dominant cost is CPU cycles spent in the lock and syscall overhead.

Benchmarking Memory Tax

Monitoring Memory Usage with System Tools

• Resident Set Size (RSS): ps -o rss,pid,comm -p <scraper-pid>.
• Page faults: pidstat -p <scraper-pid> -r 1 reports minor/major faults.
• Memory bandwidth: likwid-perfctr -g MEM -c 2 -t 10s (if likwid installed) or perf stat -e mem_load_retired.l3_miss,mem_load_retired.l1_miss.

Running Memory‑Intensive Benchmarking Tests

To stress memory, increase the amount of data scraped per poll:

• Add per‑queue statistics for all 16 TX and 16 RX queues (32 counters × 8 bytes = 256 bytes per poll).
• Optionally read extended ethtool stats (ethtool -S eth0) which can return > 1 KB of counters.
• Run the same frequency sweep while monitoring RSS.

Analyzing Memory Usage Patterns and Identifying Limitations

• RSS growth: stays constant (~2 MiB) regardless of poll rate because the scraper does not allocate per‑sample buffers; it reuses a fixed stack array.
• Page faults: remain at zero after startup; no dynamic allocation → no fault pressure.
• Memory bandwidth: at 10 kHz polling of 1 KB per poll → 10 MB/s, far below the ~20 GB/s DDR4 bandwidth of the test machine. No measurable impact on overall memory utilization.
• Conclusion: Memory tax is insignificant compared to CPU tax for this workload; the limiting factor is CPU cycles spent in locking and syscall overhead, not memory bandwidth.

Troubleshooting Common Issues

Identifying and Resolving CPU‑Related Bottlenecks

Symptom: Rising %softirq on NAPI cores, increased packet loss, latency jitter.

Diagnosis:

• perf top shows __u64_stats_sync and vfs_read as hot.
• pidstat shows high voluntary context switches on the scraper core.

Resolution:

• Batch reads: Open the directory once and use getdents/readv to fetch all statistics/* entries, reducing path lookups.
• Use netlink: RTM_GETLINK with IFLA_STATS64 returns a single struct containing all counters, eliminating per‑counter syscalls and the seqlock.
• Increase poll interval to stay below measured CPU headroom (e.g., < 5 % of a core).
• Pin scraper to a core isolated from NAPI and enable nohz_full to reduce tick interference.
• Consider eBPF map: Attach an XDP program that increments a per‑CPU map; the scraper reads the map via bpf_map_lookup_elem (still a syscall but lock‑free).

Identifying and Resolving Memory‑Related Issues

Symptom: Unexpected RSS growth or OOM killer activation.

Diagnosis: Check pmap <pid> for anonymous allocations; look for leaks in the scraper (e.g., failure to free() buffers).

Resolution:

• Ensure all buffers are stack‑allocated or freed after use.
• If using libnl or libmnl, verify that objects are released (nlmsg_free, cache_free).
• Use valgrind --leak-check=full during development.

CLI Examples to Troubleshoot and Optimize Performance

# 1. Verify per‑core load while scraper runs at 5 kHz
mpstat -P ALL 1 | awk '$3=="2"{print $0}'   # core 2 only

# 2. Identify hot syscalls
perf record -p $(pgrep scraper) -g sleep 10
perf report --stdio | grep -E "sys_read|vfs_read|__u64_stats_sync"

# 3. Switch to netlink‑based scraper and compare
taskset -c 2 ./scraper_netlink -i 200 -i eth0 &