What unknown-unicast flooding really costs

Introduction to Stale FDB State and Its Impact

Understanding FDB State and Virtual Switches

A Forwarding Database (FDB) maps MAC addresses to egress ports in a layer‑2 switching domain. In a virtual switch (vSwitch) such as Open vSwitch (OVS), Linux bridge, or a DPDK‑based vSwitch, the FDB is populated dynamically as frames are learned (source MAC → ingress port) or statically configured. The FDB enables the vSwitch to forward unicast frames directly to the known port instead of flooding.

When the FDB entry for a destination MAC becomes stale—i.e., the stored port no longer matches the actual location of that MAC—the vSwitch cannot determine the correct egress port. According to IEEE 802.1D bridging rules, the frame must be flooded to all ports in the broadcast domain (except the ingress port). In a dense east‑west topology (many VMs or containers communicating peer‑to‑peer), a single stale entry can cause a large fraction of traffic to be sent out every port, multiplying the workload on the vSwitch’s data plane.

Effects of Stale FDB State on Network Performance

Stale FDB state triggers three observable performance degradations:

1. CPU overhead – Each flooded frame requires the vSwitch to perform a lookup, miss, and then execute the flood path (often a broadcast to all local vhost‑user/virtio ports). The per‑packet cost rises from a single hash lookup + direct transmit to a lookup + loop over N‑1 egress queues.
2. Queueing pressure – The transmit queues of all flooded ports receive duplicate copies of the same frame. If the egress links are not over‑provisioned, queues build up, increasing back‑pressure and potentially causing packet drops.
3. Tail‑latency inflation – Flooded frames experience variable queuing delay depending on the instantaneous load on each egress port. The latency distribution develops a heavy tail; the 99th‑percentile (or higher) latency can increase by an order of magnitude while the median stays relatively unchanged.

Quantifying these effects requires a repeatable test that can:

• Inject a controllable amount of stale FDB state,
• Generate a steady east‑west traffic matrix,
• Measure CPU utilization, queue depths, and latency percentiles with minimal observer overhead.

The sections below describe a full‑featured test harness built on Linux OVS (v2.17+), QEMU/KVM VMs, and the MoonGen traffic generator (or DPDK test‑pmd as an alternative). All commands are given for Ubuntu 22.04 LTS with Linux 6.5 kernel; adjust package names for other distros.

Setting Up the Test Environment

Hardware and Software Requirements

All commands assume sudo privileges. Install the basics:

sudo apt-get update
sudo apt-get install -y openvswitch-switch qemu-kvm libvirt-daemon-system \
    iproute2 ethtool perf linux-tools-common moon-gen

Configuring the Virtual Switch and Network Topology

We build a star topology: a single OVS bridge (br0) with N host‑side vhost‑user ports attached to QEMU VMs, plus a single representor port (pf0vf0) connected to the traffic generator NIC. The bridge runs in netdev datapath mode for optimal packet‑per‑second performance.

1. Create the bridge

sudo ovs-vsctl add-br br0
sudo ovs-vsctl set bridge br0 datapath_type=netdev
sudo ovs-vsctl set bridge br0 other-config:disable-in-band=true
sudo ovs-vsctl set bridge br0 fail_mode=secure

2. Add the uplink representor (assumes NIC ens1f0 with VF 0 bound to the driver)

# Bind VF to vfio-pci (required for DPDK/OVS netdev)
sudo modprobe vfio-pci
sudo echo "0000:03:00.0" > /sys/bus/pci/devices/0000:03:00.0/driver/unbind
sudo echo "vfio-pci" > /sys/bus/pci/devices/0000:03:00.0/driver_override
sudo echo "0000:03:00.0" > /sys/bus/pci/devices/0000:03:00.0/driver/bind

# Add representor to OVS
sudo ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk \
    options:dpdk-devargs=0000:03:00.0,representor=[0]

3. Create VMs with vhost‑user ports

For each VM i (0 ≤ i < N):

# Create a vhost-user socket directory
sudo mkdir -p /var/run/openvswitch/vhost-user
sudo chown $USER:$USER /var/run/openvswitch/vhost-user

# Add vhost-user port to OVS
sudo ovs-vsctl add-port br0 vhost-user$i \
    -- set Interface vhost-user$i type=dpdkvhostuser \
    options:vhost-server-path=/var/run/openvswitch/vhost-user/vhost-user$i.sock

# Launch QEMU VM (example with 2 vCPUs, 2 GB RAM, virtio-net-pci)
sudo qemu-system-x86_64 \
    -enable-kvm -cpu host -smp 2 -m 2048 \
    -object memory-backend-file,id=mem,size=2048M,mem-path=/dev/hugepages,share=on \
    -numa node,memdev=mem -mem-prealloc \
    -chardev socket,id=char0,path=/var/run/openvswitch/vhost-user/vhost-user$i.sock \
    -netdev type=vhost-user,id=net0,chardev=char0,vhostforce \
    -device virtio-net-pci,mac=52:54:00:12:34:$(printf "%02x" $i),netdev=net0 \
    -drive file=/var/lib/libvirt/images/ubuntu-22.04.qcow2,if=none,id=hd0,format=qcow2 \
    -device virtio-blk-pci,drive=hd0 -nographic &

Repeat the above for the desired number of VMs (e.g., N = 24). All VMs should be on the same subnet (e.g., 10.0.0.0/24) and have a static IP configured inside the guest (e.g., 10.0.0.10+i). Verify connectivity with ping before proceeding.

Implementing Stale FDB State Scenarios

Staleness can be introduced in two controllable ways:

A. Aging‑time reduction + traffic pause

1. Set a very short FDB aging time (e.g., 1 second) on the bridge:

sudo ovs-vsctl set bridge br0 other-config:max-age=1

2. Stop all east‑west traffic for a period longer than the aging time (e.g., 5 seconds). OVS will then purge learned entries.
3. Restart traffic; the first packet for each destination will miss the FDB and cause a flood until the reply populates the entry again.

B. Manual FDB flush + static MAC mis‑placement

1. Flush the dynamic FDB:

sudo ovs-appctl fdb/flush br0

2. Add a static FDB entry that points to the wrong port (e.g., bind MAC 52:54:00:12:34:56 to dpdk0 while the VM actually resides on vhost-user5):

sudo ovs-appctl fdb/add br0 52:54:00:12:34:56 dst=dpdk0

3. Ensure the VM’s actual MAC is different (or change the VM’s MAC inside the guest) so that the static entry is now stale. Any packet destined for that MAC will be flooded.

For a repeatable test we recommend Option B because it lets us control exactly how many MACs are stale and which ports they point to, enabling a deterministic load profile.

Quantifying CPU Impact

Monitoring CPU Utilization

CPU impact is best observed at three granularities:

• Host‑level (top, pidstat, /proc/stat) to see total vSwitch and VM usage.
• Per‑thread (perf top, pidstat -t) to identify whether the OVS handler threads or the vhost‑user loops are the hotspot.
• Softirq / NAPI (cat /proc/softirqs) to measure packet‑processing overhead.

Measuring CPU Usage with CLI Tools

1. Baseline measurement (no stale FDB) – Run a steady traffic stream (e.g., 10 Gbps bidirectional) for 30 seconds and capture CPU stats:

# Start traffic generator (MoonGen) in background
sudo ./build/MoonGen -l 0-3 -n 4 --pcap ./pcap/udp_64.pcap --tx-rate 10g --duration 30s &
MG_PID=$!

# Collect host CPU usage every second
pidstat 1 30 > cpu_baseline.log

# Wait for traffic to finish
wait $MG_PID

2. Stale‑FDB measurement – Insert stale entries as described, then repeat the same collection:

# Insert stale MACs (example: 10 stale MACs pointing to dpdk0)
for i in {0..9}; do
    sudo ovs-appctl fdb/add br0 02:00:00:00:00:$(printf "%02x" $i) dst=dpdk0
done

# Repeat traffic and measurement
sudo ./build/MoonGen -l 0-3 -n 4 --pcap ./pcap/udp_64.pcap --tx-rate 10g --duration 30s &
MG_PID=$!
pidstat 1 30 > cpu_stale.log
wait $MG_PID

3. Compute delta – Use awk to average the %CPU column for the OVS process (ovs-vswitchd):

awk '/ovs-vswitchd/ {sum+=$8; count++} END {print "avg CPU %:", sum/count}' cpu_baseline.log
awk '/ovs-vswitchd/ {sum+=$8; count++} END {print "avg CPU %:", sum/count}' cpu_stale.log

Example Code for CPU Monitoring (Python + psutil)

For longer runs or automated plotting, a short Python helper can sample CPU usage of the OVS daemon and the VMs:

#!/usr/bin/env python3
import psutil, time, csv, argparse

def monitor(pid_file, interval=1, duration=30):
    with open(pid_file) as f:
        target_pid = int(f.read().strip())
    proc = psutil.Process(target_pid)
    with open('cpu_trace.csv', 'w', newline='') as csvfile:
        writer = csv.writer(csvfile)
        writer.writerow(['timestamp', 'cpu_percent'])
        start = time.time()
        while time.time() - start < duration:
            cpu = proc.cpu_percent(interval=interval)
            writer.writerow([time.time(), cpu])
            time.sleep(interval)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('--pid-file', required=True, help='File containing ovs-vswitchd PID')
    parser.add_argument('--interval', type=float, default=1.0)
    parser.add_argument('--duration', type=int, default=30)
    args = parser.parse_args()
    monitor(args.pid_file, args.interval, args.duration)

Run it alongside the traffic generator:

# Save ovs-vswitchd PID
pgrep ovs-vswitchd > ovs.pid
python3 monitor_cpu.py --pid-file ovs.pid --interval 0.5 --duration 60 &

The resulting CSV can be fed into pandas or Grafana for trend analysis.

Assessing Queueing Impact

Understanding Queueing Mechanisms in Virtual Switches

OVS netdev datapath uses DPDK rings for each port’s transmit and receive queues. When a frame is flooded, the same mbuf is reference‑counted and enqueued into each egress ring. If the NIC’s TX queues cannot keep up (due to line‑rate limits or insufficient descriptors), the DPDK ring fills, causing the rte_ring_enqueue_bulk call to return -ENOSPC. OVS then backs off, increments the port’s tx_drop counter, and eventually applies back‑pressure to the ingress side via flow control or packet drops.

Key metrics to watch:

Measuring Queue Depths and Latency

We will sample the DPDK TX ring occupancy for the uplink (dpdk0) and a few vhost‑user ports (vhost-user0…vhost-userN-1). OVS exposes these via the dpif-netdev/pmd-stats-show command.

# Helper to extract TX ring usage for a given port
get_tx_usage() {
    local port=$1
    ovs-appctl dpif-netdev/pmd-stats-show | \
        awk -v p="$port" '$0 ~ p {for(i=1