Your Traffic Generator Placement Changes the Answer

Introduction

Traffic generation can be colocated with the device under test (DUT) on the same host, placed in the same rack, or located remotely. As the distance increases, the generator’s load becomes more isolated from the DUT, but the path introduces additional latency, jitter, and potential loss. In virtual forwarding benchmarks the first ceiling is often the generator’s ability to inject packets, not the dataplane’s forwarding capacity.

Same‑Host Traffic Generation

The generator and DUT share a compute node (VM, container, or bare‑metal). Traffic typically uses a virtual interface pair (veth, macvlan, Linux bridge) or the loopback interface. Because they compete for the same CPU, memory subsystem, and NIC queues, any limitation in the generator’s packet‑creation path directly caps the measured forwarding capacity.

Methodology

1. Interface selection – Create a veth pair (veth0 ↔ veth1) or bind a macvlan interface to the host NIC.
2. Network stack – Assign IP addresses to each end; the DUT (e.g., FRR) listens on one side, the generator transmits on the other.
3. Packet generation – Use tools that can drive the NIC or kernel transmit path:
   • iperf3 for TCP/UDP bulk throughput.
   • pktgen (kernel module) for configurable rates, sizes, bursts.
   • MoonGen or TRex (DPDK‑based) for high‑rate, CPU‑isolated generation.
4. CPU affinity – Pin the generator to a dedicated core (taskset or numactl) to avoid interference with DUT forwarding threads.
5. Measurement – Capture byte counters on the NIC (ethtool -S <iface>) or inside the DUT (show interface in FRR).

Same‑Rack Traffic Generation

Generator and DUT reside on two distinct physical servers in the same rack, linked via a top‑of‑rack (ToR) switch. Traffic traverses at least one L2 hop and uses the physical NICs of both servers. Latency is low (sub‑µs to a few µs), but the generator now contends for NIC transmit queues, interrupt handling, and PCIe bandwidth independent of the DUT.

Methodology

1. Physical connectivity – Connect servers to the same ToR switch via 10/25/40/100 GbE ports.
2. IP addressing – Assign /30 or /31 subnets to the point‑to‑point links; optionally run OSPF/BGP to verify reachability.
3. Generator choice – Prefer DPDK‑based generators (TRex, MoonGen) to bypass the kernel and saturate a 25 GbE NIC with minimal CPU overhead. If kernel tools are used, enable SO_REUSEPORT and increase txqueuelen.
4. NIC tuning – Disable interrupt coalescing (ethtool -C <iface> rx-usecs 0 tx-usecs 0), set RSS to spread flows across queues, and enable LRO only if per‑packet timestamps are not required.
5. Isolation – Reserve a NUMA node for the generator (numactl --cpunodebind=0 --membind=0) and another for the DUT to avoid cross‑node memory traffic.
6. Validation – Verify lossless transmission with ping -f and measure latency with hping3 -i u1000.

Remote‑Host Traffic Generation

Generator and DUT are in different racks, data centers, or across a WAN emulator. Traffic crosses multiple switches/routers, incurring higher latency (tens of µs to ms) and jitter. The generator’s load is isolated from the DUT’s compute resources, but the path adds buffering, queuing, and possible packet loss.

Methodology

1. Path emulation – Insert a latency/jitter emulator (netem on an intermediate Linux box) or a hardware WAN emulator to inject desired RTT (e.g., 50 ms) and jitter.
2. Generator placement – Locate the generator on a dedicated traffic‑generation appliance or a server with a SmartNIC (e.g., Mellanox BlueField) to offload packet generation.
3. Protocol considerations – For TCP benchmarks, enable congestion control suited to high‑BDP links (bbr or westwood+). For UDP, adjust packet size to avoid fragmentation (MTU discovery).
4. Rate limiting – Use traffic shaping (tc qdisc add dev <iface> root tbf rate 10gbit burst 1542 latency 50ms) to prevent the generator from oversubscribing the emulated link, which would mask the DUT’s forwarding limit.
5. Monitoring – Collect end‑to‑end metrics with tcpdump -w and post‑process with tcptrace or Wireshark to verify retransmits and out‑of‑order delivery.

Benchmarking Virtual Forwarding

Tools

Benchmark Scenarios

Each scenario runs for a fixed duration (e.g., 60 s) and is repeated three times to obtain a mean and 95 % confidence interval. CPU utilization is sampled with pidstat 1 and memory with vmstat 1.

Identifying the Generator as the First Ceiling

Signs the Generator Is the Bottleneck

• CPU saturation – top or pidstat shows the generator process > 90 % CPU while DUT forwarding threads stay < 30 %.
• NIC transmit queue stalls – ethtool -S <iface> reports rising tx_queue_*_stopped or tx_timeout counters, indicating the NIC cannot keep up with the generator’s descriptor ring.
• Packet loss at the source – pktgen or TRex shows increasing tx_drop/tx_err counters while the DUT reports zero inbound drops.
• Flattening throughput curve – Raising offered load (more streams, higher packet rate) yields no increase in measured forwarding rate; latency rises sharply.
• Interrupt overload – /proc/interrupts shows a single NIC IRQ consuming > 80 % of a core’s time, despite the DUT using multiple queues for receive.

When these symptoms appear, the dataplane (virtual switch, OVS, or FRR forwarding path) is still under‑utilized; the first ceiling is the generator’s ability to feed packets into the network.

Troubleshooting Generator‑Related Bottlenecks

CLI Commands

# 1. Verify CPU affinity and utilization
pidstat -p $(pgrep -f trex) 1

# 2. Check NIC transmit queue drops
ethtool -S eth0 | grep -E 'tx_queue.*_stopped|tx_timeout'

# 3. Inspect interrupt distribution
cat /proc/interrupts | grep eth0

# 4. Look at packet generation stats (TRex example)
sudo trex-console -c show stats

# 5. Kernel pktgen interface (if using pktgen)
cat /proc/net/pktgen/kpktgend_0

Code Examples

TRex STL script – core pinning and large bursts

from trex.stl.api import *

class STLMix1(object):
    def get_stream(self, direction=0, **kwargs):
        # 64‑byte UDP packet, MAC swap per direction
        pkt = Ether()/IP(src="10.0.0.1", dst="10.0.0.2")/UDP(dport=1234, sport=5678)/Raw(b'x'*18)
        return STLStream(packet=STLPkt(pkt=pkt),
                         mode=STLTXCont(pps=10_000_000))  # 10 Mpps

    def get_streams(self, direction=0, **kwargs):
        return [self.get_stream(direction=direction)]

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser(description='TRex STL streams')
    parser.add_argument('-c', '--core', type=int, default=2,
                        help='Core to pin TRex')
    args = parser.parse_args()
    from trex.stl.trex_stl_client import STLClient
    client = STLClient()
    client.connect()
    client.acquire(force=True)
    client.set_service_mode(cores=args.core)   # pins TRex to core 2
    client.add_streams(STLMix1().get_streams(), ports=[0])
    client.start(ports=[0], mult="10s")
    client.wait_on_traffic(ports=[0])
    client.get_stats()
    client.disconnect()

Key optimizations

• set_service_mode(cores=X) binds the TRex DPDK process to a dedicated core, eliminating scheduler jitter.
• STLTXCont(pps=…) avoids per‑packet Python overhead; the rate is enforced in the DPDK TX loop.
• Payload is a static Raw block to avoid dynamic memory allocation inside the fast path.

Kernel pktgen – reduced interrupt load

# Load pktgen module
sudo modprobe pktgen

# Create a thread per CPU core to spread the load
for i in {0..3}; do
    echo "Adding device eth0 to cpu $i"
    echo eth0 > /proc/net/pktgen/kpktgend_$i
done

# Configure each thread
for i in {0..3}; do
    DEV=/proc/net/pktgen/kpktgend_$i
    echo "count 0" > $DEV               # run forever
    echo "clone_skb 1" > $DEV           # clone skb per packet (cheap)
    echo "pkt_size 64" > $DEV
    echo "dst_mac 00:11:22:33:44:55" > $DEV
    echo "dst_ip 10.0.0.2" > $DEV
    echo "dst_port 9" > $DEV            # discard port
    echo "flag QUEUE_MAP_CPU" > $DEV    # map queue to CPU
    echo "delay 0" > $DEV               # no inter‑packet delay (max rate)
done

# Start generation
echo "Start" > /proc/net/pktgen/pgctrl

Explanation – By creating a pktgen thread per core and enabling QUEUE_MAP_CPU, each core services a separate transmit queue, reducing per‑core interrupt pressure and allowing the NIC to sustain higher line‑rate before the generator becomes the limiter.

Scaling Limitations of Traffic Generators

Same‑Host Scaling Limitations

• CPU bound – Generator shares cores with the DUT; increasing packet rate steals cycles from forwarding threads.
• Memory bandwidth – Large packets (e.g., 9 KB jumbo frames) can saturate the memory subsystem before the NIC is saturated.
• Virtualization overhead – Inside a container or VM, the veth pair adds an extra copy‑on‑write and checksum step, limiting achievable pps to roughly 2‑3 Mpps on a modern x86 core (observed with FRR + pktgen).

Same‑Rack Scaling Limitations

• NIC transmit queue depth – Default queue size (256 descriptors) can cause head‑of‑line blocking at > 10 Mpps for 64‑byte packets; increasing tx ring size via ethtool -G eth0 tx 4096 helps but consumes more host memory.
• PCIe bandwidth – A single x8 Gen3 slot offers ~8 GB/s; saturating a 25 GbE NIC (≈3 GB/s) leaves headroom, but dual‑port 100 GbE cards can exceed the PCIe capacity of older platforms, making the NIC the bottleneck before the generator CPU.
• NUMA effects – If the generator and NIC reside on different NUMA nodes, cross‑node memory traffic adds latency and reduces achievable pps.

Remote‑Host Scaling Limitations

• Path emulation overhead – netem adds per‑packet queuing in the kernel; at high rates (> 5 Mpps) the emulator itself becomes the limiter, showing increased latency and jitter.
• SmartNIC offload limits – While a SmartNIC can generate traffic at line rate, its internal traffic‑generation engines have finite throughput; exceeding that limit results in drops or back‑pressure.
• WAN emulator capacity – Hardware WAN emulators have a maximum packet‑processing rate; surpassing it introduces additional loss and distortion that masks the DUT’s true forwarding limit.

By recognizing these symptoms and applying the tuning steps above, you can isolate whether the traffic generator or the virtual dataplane is the first performance ceiling in your forwarding benchmarks.