Sizing AF_XDP rings for short overload not averages

Introduction to Ring Budgets

Definition and Importance of Ring Budgets

In a Linux networking dataplane a ring (also called a descriptor queue) is a fixed‑size circular buffer that holds pointers to packet buffers (SKBs, XDP buffers, or DMA descriptors). The NIC writes received packets into the Rx ring, the driver posts transmit descriptors into the Tx ring, and the NIC writes completion statuses into the Completion (or TX‑completion) ring. A fill ring is used by some drivers (e.g., virtio‑net, vhost‑user) to hold empty buffers that the NIC can fill.

The budget of a ring is the minimum number of descriptors required to guarantee that, under a given traffic pattern, the producer (NIC or driver) never outruns the consumer (driver or application) and therefore no packets are dropped due to ring overflow. Correctly sizing these budgets is essential for:

• Zero‑loss operation at line rate for bursty workloads (e.g., RPC, financial trading, HPC).
• Predictable latency – a ring that is too small causes back‑pressure and queuing jitter; a ring that is too large wastes memory and can increase cache‑miss latency.
• CPU efficiency – oversized rings increase the cost of descriptor prefetch and cleanup; undersized rings force the driver into frequent interrupt handling or busy‑poll loops.

Current Limitations of Average PPS‑Based Models

Traditional ring‑budget calculators rely on an average packets‑per‑second (PPS) figure:

budget_avg = average_pps * target_latency

where target_latency is the maximum tolerable queuing delay (often expressed in seconds). This approach assumes a Poisson or smooth traffic arrival process. It fails when:

Consequently, systems designed with the average PPS method often experience unexpected packet loss under realistic, bursty traffic, prompting over‑provisioning of rings (wasting memory) or under‑provisioning (causing drops).

Deriving Ring Budgets from Burst Width and Service Time

Theoretical Background and Formulas

We define three observable quantities:

• Burst width (W) – the maximum duration (seconds) over which the NIC can receive packets at line rate without an idle gap. Equivalently, the maximum number of back‑to‑back packets that can arrive is N_burst = W * R_line, where R_line is the line‑rate in packets per second (pps).
• Service time (S) – the average time (seconds) required for the driver to consume one descriptor from the ring and make it available for reuse. This includes DMA completion, interrupt handling (or polling), SKB allocation/free, and any XDP/eBPF processing.
• Line‑rate packet spacing (τ) – the inverse of line‑rate: τ = 1 / R_line. It is the nominal time between two successive packets when the link is saturated.

During a burst of width W, the NIC can place at most N_burst = W / τ descriptors into the ring. In the same interval the driver can consume at most W / S descriptors (because each consumes S seconds). The backlog that must be absorbed by the ring is therefore:

backlog = max(0, N_burst - W / S)
        = max(0, W * (1/τ - 1/S))

If S <= τ (the driver can keep up with line rate), the backlog is zero and a minimal ring (often just a few descriptors) suffices. When S > τ (the driver is slower than the line rate), the ring must hold the excess descriptors.

To provide a safety margin for jitter in service time (e.g., occasional cache misses, interrupt latency) we multiply by a safety factor α ≥ 1. The ring budget (B) in descriptors is:

B = ceil( α * W * (1/τ - 1/S) )

When 1/τ - 1/S is negative (driver faster than line rate) we set B = 0 and then add a minimal baseline (e.g., 4 descriptors) to avoid empty rings.

The same formula applies to each ring type, but the service time differs:

Thus we compute four budgets: B_fill, B_rx, B_tx, B_comp.

Calculating Fill, Rx, Tx, and Completion Ring Budgets

Step‑by‑step procedure

1. Measure or configure line rate (R_line). For a 10 GbE link with 64‑byte Ethernet frames (including preamble, IFG) the theoretical max is:

   R_line = (10 × 10^9 bits/s) / ( (64+20+12) × 8 ) ≈ 14.88 Mpps

   Adjust for actual frame size (L2_len) if known.

2. Determine burst width (W) from traffic profile or measurement:

   • Use a traffic generator capable of programmable burst patterns (e.g., MoonGen, pktgen with burst mode).
   • Capture with tcpdump -i <iface> -w burst.pcap and compute the longest interval with inter‑packet gap < τ.
   • Alternatively, set W as a design parameter (e.g., 10 µs for microburst tolerance).

3. Measure service time (S) for each ring:

   • Use perf record -e cycles -g -p <pid> while running a steady‑state traffic load.
   • Derive average cycles per descriptor, convert to seconds using CPU frequency.
   • For more precision, attach a BPF tracepoint to the driver’s descriptor‑handling functions (e.g., ixgbe_clean_rx_irq, ixgbe_tx_timeout) and timestamp entry/exit.

4. Pick safety factor (α):

   • Start with α = 1.2 (20 % headroom) and increase if drops are observed under bursty traffic.
   • Account for worst‑case interrupt latency (e.g., 10 µs on a tickless kernel) by adding α_int = latency_int / S.

5. Apply the formula and round up to the next power of two (most NICs require ring sizes to be powers of two).

Example calculation (10 GbE, 84‑byte frames, W = 5 µs, S_rx = 800 ns, α = 1.2):

τ = 1 / R_line
R_line = 10e9 / ((84+20+12)*8) = 10e9 / (116*8) ≈ 10.75 Mpps
τ ≈ 93 ns

1/τ - 1/S = (1/93e-9) - (1/800e-9) ≈ 10.75e6 - 1.25e6 = 9.5e6 packets/s
B = ceil(1.2 * 5e-6 * 9.5e6) = ceil(1.2 * 47.5) = ceil(57) = 57 descriptors
Next power of two => 64 descriptors

Thus the Rx ring should be at least 64 descriptors to absorb a 5 µs line‑rate burst when the driver needs ~800 ns per packet.

Comparison with Average PPS‑Based Models

Assume we measure an average throughput of 2 Mpps over a 1‑second interval (including idle periods). With a target latency of 10 µs:

budget_avg = 2e6 * 10e-6 = 20 descriptors

Our burst‑aware calculation gave 64 descriptors – more than three times larger. The average model would severely underestimate the needed buffer for the same traffic pattern, leading to drops during the microburst. Conversely, if the traffic is truly smooth, the burst‑aware model will converge to a similar size (because W will be small or S will be close to τ).

Implementation and Testing

Setting Up a Test Environment

Kernel configuration snippet (add to /etc/sysctl.conf or via sysctl -w):

net.core.netdev_budget = 600          # NAPI budget per poll
net.core.netdev_budget_usecs = 0      # disable time budget, rely on packet count

Code Examples for Calculating Ring Budgets

Below is a self‑contained Python 3 script that reads NIC parameters, computes the four ring budgets, and prints the recommended power‑of‑two sizes.

#!/usr/bin/env python3
import math
import argparse

def line_rate_pps(speed_gbps: float, frame_len: int) -> float:
    """
    Compute theoretical line rate in packets per second.
    speed_gbps: link speed in Gbit/s (e.g., 10)
    frame_len:  L2 frame length in bytes (including Ethernet header, no FCS)
    The constant 20 accounts for preamble (7 bytes) + SFD (1 byte) + IFG (12 bytes).
    """
    # Ethernet preamble + SFD + inter‑frame gap = 20 bytes
    overhead = 20
    bits_per_packet = (frame_len + overhead) * 8
    return (speed_gbps * 1e9) / bits_per_packet

def next_power_of_two(x: int) -> int:
    return 1 if x == 0 else 1 << (x - 1).bit_length()

def ring_budget(burst_width_s: float, service_time_s: float,
                line_rate_pps_val: float, safety_factor: float = 1.2,
                min_descriptors: int = 4) -> int:
    """
    Compute ring budget in descriptors.
    Returns at least `min_descriptors` to avoid empty rings.
    """
    tau = 1.0 / line_rate_pps_val
    backlog = burst_width_s * (1.0 / tau - 1.0 / service_time_s)
    if backlog < 0:
        backlog = 0.0
    budget = math.ceil(safety_factor * backlog)
    return max(min_descriptors, budget)

def main():
    parser = argparse.ArgumentParser(
        description="Compute ring burst‑aware budgets for Rx/Tx/Fill/Completion."
    )
    parser.add_argument("--speed", type=float, required=True,
                        help="Link speed in Gbit/s (e.g., 10 for 10GbE)")
    parser.add_argument("--frame-len", type=int, required=True,
                        help="L2 frame length in bytes (no FCS)")
    parser.add_argument("--burst-width", type=float, required=True,
                        help="Burst width in seconds (e.g., 5e-6 for 5µs)")
    parser.add_argument("--