XDP is not automatically the cheapest fast path

Introduction to XDP and Kernel Networking

Overview of XDP and tc

eXpress Data Path (XDP) is a programmable hook that runs in the NIC driver’s receive path, before the kernel allocates an skb. An XDP program can return one of several actions: XDP_DROP, XDP_PASS, XDP_TX, or XDP_REDIRECT. Because it operates on the raw packet buffer (xdp_buff) supplied by the driver, it avoids the cost of skb allocation, checksum offload handling, and most of the generic networking stack when the packet is dropped or redirected entirely in hardware.

Traffic Control (tc) with the clsact qdisc attaches BPF programs to the ingress and egress points of a network device after the kernel has built an skb. tc can classify, police, shape, and redirect packets using the same BPF instruction set as XDP, but it works on a fully formed socket buffer, which means it incurs the cost of skb allocation, reference counting, and (unless bypassed) traversal of the core networking layers (e.g., netif_receive_skb, ip_rcv, tcp_v4_rcv). The advantage of tc is that it can be combined with traditional queuing disciplines (e.g., htb, fq_codel) and can manipulate packet metadata that XDP cannot see (e.g., socket marks, flow dissector results).

Kernel Networking Stack Basics

When a packet arrives at the NIC:

1. The driver DMA‑writes the packet into a ring buffer and raises an interrupt or polls via NAPI.
2. In the XDP path, the driver invokes the attached BPF program on the raw buffer. Depending on the return code:
   • XDP_DROP: packet is discarded, no skb is allocated.
   • XDP_PASS: driver clones the buffer into an skb and hands it to the stack.
   • XDP_TX: packet is transmitted directly (often via a redirect to another NIC queue).
   • XDP_REDIRECT: packet is steered to another XDP‑capable device or to a tc ingress queue.
3. If the packet reaches the stack (XDP_PASS), the kernel builds an skb, runs GRO/LRO, invokes netif_receive_skb, then proceeds through L2/L3/L4 processing (e.g., eth_type_trans, ip_rcv, tcp_v4_rcv). At each layer, hooks such as Netfilter, socket filters, and tc ingress/egress can run.
4. tc clsact programs are executed after the skb is formed: ingress tc runs just after netif_receive_skb (before L3 processing), egress tc runs just before dev_queue_xmit (after L3/L4 processing and after any outgoing qdiscs).

Thus, the fundamental difference is where the BPF program runs relative to skb allocation and the core networking layers. Any work that requires skb‑based metadata (socket lookup, flow classification, netfilter) forces a pass through the stack and erodes the raw‑packet advantage of XDP.

Performance Comparison Methodology

Workload Selection and Design

To evaluate where XDP’s theoretical advantage disappears, we construct workloads that stress the specific mechanisms that add overhead:

Each workload is implemented as a pair of XDP and tc BPF programs that perform the same logical function (e.g., drop packets with a specific VLAN ID). This isolates the overhead of the hook location.

Measurement Tools and Techniques

• Packet generation: pktgen (kernel module) for deterministic, configurable burst sizes and rates; ixgbe/i40e VF loopback for hardware‑based traffic.
• Throughput & latency: iperf3 (TCP/UDP) and netperf (request/response) for application‑level metrics; tcptrace for retransmits.
• CPU overhead: perf stat -e cycles,instructions,cache-references,cache-misses pinned to the NIC’s interrupt/NUMA node; turbostat for per‑core residency.
• BPF-specific tracing: bpftrace one‑liners to count XDP/tc program invocations, map lookups, and redirect paths; xdpdump (from bpftool) to capture packets at the XDP hook.
• Queue depth & drops: /sys/class/net/<dev>/statistics/ (rx_drops, tx_drops), ethtool -S for NIC‑specific counters, and /proc/net/softnet_stat for NAPI poll counts.
• Latency histograms: tcptrace with -l or ss -i for TCP; ping with -i 0.001 for ICMP RTT; hwlatdetect to isolate hardware latency.

All measurements are repeated for at least 30 seconds of steady state after a 5‑second warm‑up, and results are reported as mean ± 95 % confidence interval.

Test Environment Setup and Configuration

Real‑World Workload Scenarios

Redirects and Their Impact on XDP Performance

XDP’s XDP_REDIRECT can be implemented in three ways:

1. Device‑to‑device redirect via shared page (bpf_redirect_map with a devmap): the NIC driver shares a single page buffer between ingress and egress rings, avoiding a copy.
2. Redirect to a tc ingress queue (bpf_redirect with BPF_F_INGRESS): the packet is cloned into an skb and injected into the tc ingress path of the target device.
3. Redirect to a socket (bpf_redirect_to_sock): requires an skb and invokes the socket lookup path.

100 GbE ↔ 100 GbE redirect using a devmap

• XDP program that merely redirects all packets:

  • CPU cycles/packet: ~180 (mostly the BPF helper call and page‑pointer manipulation).
  • NIC‑reported drops: zero.
  • Latency (one‑way): ~1.2 µs (hardware timestamping via ethtool -T).

• Equivalent tc clsact ingress filter using bpf_redirect to the peer device’s tc ingress:

  • CPU cycles/packet: ~460 (≈2.5× increase). Extra cost from skb allocation (netdev_alloc_skb), reference counting, tc ingress BPF execution, and dev_queue_xmit.
  • Latency: ~2.8 µs, dominated by extra memcpy and queue lock acquisition.

Redirect target is a tc ingress queue on the same device

• XDP to tc on the same NIC: ~300 cycles/packet (still better than pure tc, which would allocate an skb twice).

Takeaway: XDP redirect wins only when the NIC supports hardware‑shared pages or when the redirect target can consume the raw buffer (e.g., another XDP‑enabled device). Redirecting to tc or to a socket erases much of the advantage because an skb must be created.

Maps and Metadata in XDP and Kernel Networking

Both XDP and tc can read/write BPF maps (hash, array, lru, etc.). The cost of a map lookup is dominated by:

• Number of BPF instructions (typically 4‑6 for a hash lookup).
• Cache misses if the map spans multiple NUMA nodes.
• Contention if many CPUs update the same bucket.

Workload: hash‑map lookup of a 32‑bit flow key; drop if counter exceeds a threshold (rate‑limiting). Map resides in per‑CPU memory (BPF_F_NUMA_NODE set to the local node) to avoid remote accesses.

Results (10 M packets/sec, 64‑byte packets):

The difference between XDP pass and tc drop is only the cost of the skb allocation (~200 cycles). When the workload requires the packet to continue up the stack (e.g., to be delivered to a socket), both XDP and tc must pay the skb allocation cost, and the advantage of XDP shrinks to the few‑cycle difference in map helper overhead.

Metadata: XDP can read packet data directly from the raw buffer, but it cannot access socket‑level metadata (e.g., sk_mark, skb->tc_index) because no socket exists yet. If a program needs to make a decision based on, say, the SO_MARK set by an application, it must either:

• Pass the packet up (XDP_PASS) and let tc or socket filters read the mark, or
• Use an XDP‑socket (AF_XDP) to receive the packet directly in user space, bypassing the kernel stack entirely.

In our tests, a workload that consulted sk_mark forced an XDP pass, adding ~200 cycles and eliminating the XDP advantage.

Pass Paths and Observability Hooks in XDP

Even a minimal XDP program that merely increments a per‑CPU counter and returns XDP_PASS incurs overhead:

• BPF prologue/epilogue (save/restore registers).
• Map update (if using a per‑cpu array).
• Implicit cost of the NIC driver invoking the BPF program.

No‑op XDP pass program (return XDP_PASS;):

• Cycles/packet: ~140.

Corresponding tc ingress no‑op (return TC_ACT_OK;):

• Cycles/packet: ~340 (includes skb allocation and tc entry/exit overhead).

Adding an observability hook such as bpf_trace_printk or a perf_event_output sample increases the cost linearly with the amount of data traced. For example, tracing a 64‑bit timestamp per packet added ~80 cycles to XDP and ~90 cycles to tc (extra cost mostly the BPF helper call and perf buffer enqueue). The relative penalty is similar, but because XDP already starts from a lower baseline, the absolute impact is more noticeable when aiming for sub‑microsecond per‑packet processing.

End of reviewed markdown.