Microbursts, NAPI Budget, and Fast-Path Tradeoffs

Introduction to NAPI Budget Tuning and XDP

Overview of NAPI and XDP

The Linux networking stack uses NAPI (New API) to transition from interrupt‑driven to polling‑driven packet reception when the NIC reports a high rate of incoming frames. When an interrupt occurs, the kernel disables further interrupts for that queue and enters a NAPI poll loop, which repeatedly calls the driver’s poll method until either the budget is exhausted or no more packets are available. The two key knobs that govern this loop are:

• net.core.netdev_budget – maximum number of packets that a single NAPI poll may process (default 300).
• net.core.netdev_budget_usecs – maximum time (in microseconds) a NAPI poll may run before being pre‑empted (default 2000 µs).

XDP (eXpress Data Path) attaches a BPF program at the very earliest point in the receive path, right after the NIC DMA places a packet into the descriptor ring but before the kernel allocates an skb. XDP programs can drop, redirect, or pass the packet up the stack with virtually no per‑packet overhead beyond the BPF instruction count.

Importance of Benchmarking under Microburst Traffic

Microbursts are short, intense spikes of packet arrival that can exceed the average line rate by an order of magnitude for a few microseconds to milliseconds. They stress the transient buffering capacity of NIC rings, the NAPI polling mechanism, and any inline BPF hooks. Understanding how NAPI budget, busy‑poll, and hook placement interact under such traffic is essential for:

• Determining the point at which packet loss begins (the “loss onset”).
• Quantifying the CPU cost of softirq processing versus busy‑poll spinning.
• Identifying whether moving work from tc (traffic control) to XDP reduces ring occupancy and backlog growth.

A repeatable benchmark that measures ring occupancy, softirq time, backlog growth, and loss onset under controlled microburst patterns provides the evidence needed to tune these knobs safely.

NAPI Budget Tuning

Understanding NAPI Budget Parameters

When the NIC raises an interrupt, the kernel disables further interrupts and enters the NAPI loop. The loop continues until either:

• The packet count processed reaches netdev_budget, or
• The elapsed time exceeds netdev_budget_usecs, or
• The driver’s poll returns 0 (no more packets).

If the budget is too low, the kernel will exit NAPI early, re‑enable interrupts, and potentially incur many interrupts per second, increasing softirq overhead. If the budget is too high, a single NAPI poll may monopolize the CPU, delaying other softirqs or tasks and increasing latency.

Impact of NAPI Budget on Network Performance

• Low budget (e.g., 64 packets) → More frequent NAPI exits → higher interrupt rate → increased softirq CPU usage, but better latency for other CPUs because each poll is short.
• High budget (e.g., 1024 packets) → Fewer NAPI exits → lower interrupt overhead, but each poll can consume a large timeslice, potentially causing softirq starvation and increased packet queuing if the application cannot keep up.
• Interaction with netdev_budget_usecs – If the time budget is reached before the packet budget, the kernel yields even if packets remain, protecting against long-running polls.

Under microburst traffic, the NIC ring can fill faster than the application can drain it. A larger NAPI budget allows the kernel to pull more packets per poll, reducing the chance that the ring overflows before the next poll. However, if the application’s receive path (e.g., socket queue) is the bottleneck, pulling more packets merely increases backlog in the kernel’s skb queues, which may manifest as increased rx_drops or rx_over_errors.

Code Examples for NAPI Budget Tuning

# View current settings
sysctl net.core.netdev_budget
sysctl net.core.netdev_budget_usecs

# Temporarily increase budget to 1024 packets and 5000 µs
sysctl -w net.core.netdev_budget=1024
sysctl -w net.core.netdev_budget_usecs=5000

# Make persistent across reboots (Debian/Ubuntu)
echo "net.core.netdev_budget=1024" >> /etc/sysctl.d/99-napi.conf
echo "net.core.netdev_budget_usecs=5000" >> /etc/sysctl.d/99-napi.conf
sysctl --system

# Per‑queue override (if NIC supports multiple rx queues)
for q in /sys/class/net/eth0/queues/rx-*; do
    echo 512 > $q/weight   # sets netdev_budget for that queue
done

When testing, it is useful to capture the number of NAPI polls executed and the average packets per poll:

# Show NAPI statistics (requires kernel >= 4.14)
cat /proc/net/softnet_stat
# Columns: processed, dropped, time_squeeze, ... (see Documentation/networking/softnet-stat.txt)

Busy-Poll Choices and Their Impact

Busy-Poll Mechanism Overview

Busy‑poll is an optimization that allows a socket (or the entire NIC) to poll for packets in a tight loop instead of relying on interrupts or NAPI. When enabled, the kernel repeatedly invokes the driver’s poll method in a loop, consuming CPU cycles but potentially reducing latency and avoiding softirq context switches.

Key sysctls:

• net.core.busy_poll – default busy‑poll timeout in microseconds when a socket calls recvmsg/recvfrom with MSG_WAITALL or MSG_DONTWAIT and finds no data.
• net.core.busy_read – similar to busy_poll but used by the read syscall on packet sockets (e.g., AF_PACKET).
• Per‑socket control – via setsockopt(fd, SOL_SOCKET, SO_BUSY_POLL, &val, sizeof(val)).

When busy_poll > 0, the kernel will spin for up to that many microseconds before giving up and either blocking (if the socket is blocking) or returning EAGAIN. The spin occurs in softirq context, so it adds to the softirq CPU utilization measured in /proc/stat.

Effects of Busy-Poll on CPU Utilization and Packet Processing

• Latency reduction – By avoiding the interrupt-to-softirq handoff, the first packet after a poll can be delivered to the application sooner, often shaving tens of microseconds.
• CPU cost – Busy‑poll consumes CPU cycles continuously while waiting for packets. Under idle or low‑traffic conditions, this can waste power and increase contention with other tasks. Under saturated microburst conditions, the extra CPU may be justified if it prevents ring overruns.
• Interaction with NAPI – If busy‑poll is enabled, the kernel may still use NAPI for interrupt‑driven reception; busy‑poll only kicks in when the socket’s receive find no data. In a tight loop receiving packets, the kernel will often stay in NAPI because data is present, so busy‑poll adds little overhead. Conversely, when the application cannot keep up and the socket’s receive queue empties, busy‑poll will spin, potentially exacerbating CPU load without improving packet capture.

CLI Examples for Configuring Busy-Poll

# View current global busy-poll settings
sysctl net.core.busy_poll
sysctl net.core.busy_read

# Set a global busy-poll timeout of 50 µs (useful for low‑latency UDP recv)
sysctl -w net.core.busy_poll=50
sysctl -w net.core.busy_read=50

# Make persistent
echo "net.core.busy_poll=50" >> /etc/sysctl.d/99-busypoll.conf
echo "net.core.busy_read=50" >> /etc/sysctl.d/99-busypoll.conf
sysctl --system

# Per‑socket example (C snippet)
int val = 50; // 50 µs
setsockopt(sockfd, SOL_SOCKET, SO_BUSY_POLL, &val, sizeof(val));

# Verify effect via /proc/net/softnet_stat (look for "poll" column increase)
watch -n 0.1 cat /proc/net/softnet_stat

When benchmarking, capture the CPU time spent in softirq due to busy‑poll:

# Measure softirq time before and after enabling busy-poll
pidstat -u -p ALL 1  # look for %system (softirq) column
# Or use perf:
perf stat -e cycles,instructions,cpu-clock,task-clock -a sleep 10

Hook Placement Between XDP and tc

XDP and tc Overview

• XDP – Executes in the NIC driver’s receive path, before the kernel allocates an skb. It runs in either native mode (directly in the driver) or generic mode (after the driver but before the network stack). XDP programs are limited to a small set of helper functions and must finish within a few hundred instructions to avoid excessive per‑packet cost.
• tc (traffic control) – Operates on fully formed skbs. The clsact qdisc allows attaching BPF filters at ingress (tc filter add dev eth0 ingress ...) and egress (... egress ...). tc runs after the NIC has delivered the packet to the kernel and after any XDP processing (if XDP is in native mode). Thus, the ordering is: NIC DMA → XDP (native) → kernel allocates skb → tc ingress → stack (e.g., IPv4, UDP) → socket queues.

Hook Placement Strategies for Optimal Performance

Microburst considerations:

• If the goal is to avoid ring overruns, dropping packets as early as possible (XDP native) reduces the work the kernel must do per packet, thus lowering the NAPI budget consumption and softirq load.
• If you need to re‑inject packets (e.g., load‑balancing to another interface), XDP native XDP_REDIRECT is still cheaper than tc because it avoids skb cloning.
• If you require stateful logic that cannot fit in XDP’s limited helper set, placing the program in tc ingress may be unavoidable; however, you can still offload simple early‑drop filters to XDP to reduce the load on tc.

Code Snippets for Implementing XDP and tc Hooks

# 1. Load an XDP native program (assumes NIC supports it)
ip link set dev eth0 xdp obj /opt/xdp_drop.o sec xdp

# 2. Verify XDP attachment
ip -d link show eth0 | grep xdp
# Expected output: xdp ... id 123

# 3. Load a tc ingress BPF program (after XDP)
tc qdisc add dev eth0 clsact
tc filter add dev eth0 ingress bpf da obj /opt/tc_count.o sec tc

# 4. Verify tc filter
tc filter show dev eth0 ingress

# 5. Remove both when done
ip link set dev eth0 xdp off
tc qdisc del dev eth0 clsact

Example XDP drop program (C, compiled with clang -O2 -target bpf):

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

SEC("xdp")
int xdp_drop(struct xdp_md *ctx) {
    return XDP_DROP;   // drop every packet
}
char _license[] SEC("license") = "GPL";

Example tc ingress packet counter (C):

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, __u64);
} pkt_cnt SEC(".maps");

SEC("tc")
int tc_count(struct __sk_buff *skb) {
    __u32 idx = 0;
    __u64 *val = bpf_map_lookup_elem(&pkt_cnt, &idx);
    if (val)
        __sync_fetch_and_add(val, 1);
    return TC_ACT_OK;
}
char _license[] SEC("license") = "GPL";

When benchmarking, you can toggle each hook independently to measure its contribution to CPU usage and packet loss.

Benchmarking Under Microburst Traffic

Microburst Traffic Generation Tools and Techniques

• pktgen – Built‑in Linux traffic generator capable of configuring burst length, inter‑burst gap, and packet size via /proc/net/pktgen/.
• MoonGen – DPDK‑based high‑precision generator; can produce sub‑microsecond bursts with deterministic timing.
• traffic‑generator (e.g., IXIA, Spirent) – Commercial hardware generators for line‑rate microburst emulation.

Measurement Methodology

1. Configure the NIC – Set desired NAPI budget, busy‑poll timeout, and attach/detach XDP/tc hooks as needed.
2. Warm‑up – Run a low‑rate background traffic flow to stabilize NIC rings and kernel structures.
3. Inject microbursts – Use pktgen or MoonGen to generate bursts of configurable size (e.g., 10 µs, 100 µs, 1 ms) at a target line‑rate (e.g., 10 Gbps).
4. Collect metrics – During each test run, sample:
   • Ring occupancy (ethtool -S eth0 shows rx_ring_* counters).
   • Softir