Introduction to AF_XDP and XDP_PASS

Overview of AF_XDP

AF_XDP (Address Family XDP) is a socket interface that enables zero‑copy packet I/O between the NIC and a user‑space application. Packets are placed directly into a pre‑allocated UMEM region shared with the NIC via the XDP data path. The application interacts with the NIC through four lock‑less rings:

• Fill Ring – supplies empty descriptors to the NIC.
• Rx Ring – NIC posts filled descriptors (packet address + length).
• Tx Ring – application posts descriptors for transmission.
• Completion Ring – NIC returns Tx descriptors after DMA completes.

Because the NIC DMA writes directly into the UMEM, the application can access packet payloads without any copy from kernel space. The only overhead is the ring synchronization (typically a few nanoseconds per descriptor) and the cost of polling or interrupt handling.

AF_XDP shines when the workload is packet‑rate bound and the per‑packet work in user space is minimal (e.g., simple forwarding, load‑balancing, or sampling).

Overview of XDP_PASS

XDP (eXpress Data Path) is a hook that runs earliest in the receive path, before the kernel allocates an skb. An XDP program can return one of several actions:

• XDP_DROP – discard the packet.
• XDP_PASS – let the packet continue up the normal kernel stack (allocates an skb, invokes netdev receive, etc.).
• XDP_TX – transmit directly from the NIC.
• XDP_REDIRECT – redirect to another NIC or a BPF map.

When an XDP program returns XDP_PASS, the packet follows the traditional receive path: the NIC DMA writes into a kernel‑allocated buffer, the driver builds an skb, the packet passes through netfilter, routing, and finally arrives at a socket (e.g., AF_PACKET, UDP, or TCP). The application then reads the data via recvmsg()/read(), which incurs a copy from kernel space to user space (unless zero‑copy sockets like TCP_ZEROCOPY_RECEIVE are used, which are not applicable for generic packet sockets).

XDP_PASS is useful when you want to keep the full kernel networking stack (e.g., for connection tracking, complex routing, or when you need to use existing socket APIs) while still benefitting from early packet filtering or sampling performed in XDP.

Benchmarking Methodology

Environment Setup

All experiments were pinned to a single NUMA node; the NIC port used was bound to that node via ethtool -N eth0 rx-flow-hash udp4 sdfn and ethtool -L eth0 combined 8.

Workload Definitions

Simple Drop Workload

• Goal: Measure the cost of discarding a packet as early as possible.
• XDP program: Returns XDP_DROP for every packet.
• AF_XDP variant: Application receives a packet on the Rx ring, immediately returns the descriptor to the Fill ring (no payload access).
• Metrics: Packets per second (pps) before drop, CPU utilization (% of a core), and average latency from NIC DMA to drop decision (measured via NIC TX timestamp on a loopback port for sanity).

Sample Workload

• Goal: Measure the cost of extracting a small fixed‑size sample (e.g., first 16 bytes) and then discarding the rest.
• XDP program: Returns XDP_PASS; the packet continues up the stack where a socket reads the first 16 bytes via recvmsg(MSG_TRUNC) and then closes the socket (or simply drops the packet).
• AF_XDP variant: Application reads the first 16 bytes from the UMEM buffer, then returns the descriptor to the Fill ring (no further processing).
• Metrics: Same as Drop, plus sample extraction latency (time from DMA to completion of the 16‑byte read).

Forward Workload

• Goal: Measure the cost of receiving a packet and transmitting it unchanged on a second NIC port (same hardware).
• XDP program: Returns XDP_PASS; the packet traverses the kernel stack, is sent out via a raw AF_PACKET socket (SOCK_RAW, ETH_P_ALL) on the second port.
• AF_XDP variant: Application receives a descriptor, optionally modifies nothing, and posts the same descriptor to the Tx ring of the second port (zero‑copy forward).
• Metrics: Forwarding pps, end‑to‑end latency (NIC RX timestamp on ingress port → NIC TX timestamp on egress port), CPU utilization on both cores (RX and TX), and drop rate (should be zero).

Benchmarking Tools and Metrics

• Packet rate – measured with MoonGen (-t 10 seconds) and validated with ethtool -S eth0 counters (rx_packets, tx_packets).
• CPU utilization – gathered via pidstat -p <pid> 1 and perf stat -e cycles,instructions,cache-references,cache-misses.
• Latency – hardware timestamps enabled via ethtool -T eth0 tx_vlan_hw_insert on and rx_vlan_hw_insert on; timestamps read from the NIC descriptor and converted to nanoseconds.
• Drop rate – difference between transmitted and received packets at the generator.
• Memory overhead – size of UMEM (AF_XDP) vs. kernel skb cache (XDP_PASS) measured via /proc/meminfo and slabtop.

Each workload was run for 30 seconds after a 5‑second warm‑up, with results averaged over three repetitions. Confidence intervals (± 1 σ) are reported where relevant.

Zero‑Copy AF_XDP Benchmarking

Simple Drop Workload Results

Interpretation: The zero‑copy path saves ~0.3 µs per packet by avoiding the kernel’s skb allocation and the subsequent kfree_skb in the drop path. CPU utilization drops because the only work is ring synchronization (fill → rx → fill).

Sample Workload Results