Bridge OVS or macvlan for external ports

##Introduction
Linux offers three primary ways to attach virtual interfaces (veth, tap, etc.) to a physical NIC for lab or container workloads:

Why Benchmark?

Before declaring any attachment model “production‑like” you must quantify:

• Packet‑per‑second (PPS) ceiling – maximum rate the datapath can sustain without drops.
• One‑way latency – measured from TX timestamp on the generator to RX timestamp on the receiver, including queuing and processing.
• CPU cost – cycles per packet (or %CPU on a dedicated core) incurred by the attachment model.
• Operational caveats – features that break expectations (e.g., MAC‑learning limits, ARP proxying, offload incompatibility, stateful connection tracking).

A repeatable benchmark isolates these variables so you can decide whether the added flexibility of OVS justifies its overhead, or whether macvlan’s near‑hardware performance is worth the loss of bridging features.

Benchmarking Methodology

Test Environment

All tests run on a single socket to avoid NUMA effects; the NIC is bound to a single VF assigned to the test namespace.

Tools

• pktgen – kernel packet generator with nanosecond TX timestamps (PKTGEN_TX_TIMESTAMP).
• trex – userspace traffic generator for realistic burst patterns.
• tcptrace / tshark – latency measurement using hardware timestamps (SO_TIMESTAMPING with TX_SOFTWARE/RX_HARDWARE).
• perf stat – cycles, instructions, cache‑misses, context‑switches on packet‑processing cores.
• bpftool + tracepoint:sched:sched_switch – verify core affinity and preemption.
• ethtool -k – verify offload state (GRO, GSO, TSO, UFO, RXCSUM, TXCSUM) before each run.
• ip link show type bridge, ovs-vsctl show, ip -d link show – configuration validation.

Scenarios & Workloads

Each sweep runs for 30 seconds per rate step, collecting:

• PPS – packets transmitted vs. received (difference = drops).
• Latency – average and 99th‑percentile one‑way latency (hardware TX timestamp → software RX timestamp via SO_TIMESTAMPING).
• CPU – %CPU of the core(s) handling NAPI poll (/proc/stat per‑core) and perf stat -e cycles,instructions,cache-references,cache-misses.

All numbers are reported after subtracting the baseline VF cost (i.e., the overhead introduced by the attachment model).

Linux Bridge

Configuration

# Create namespace
ip netns add testns
# Move VF into ns
ip link set dev eth0 vf 0 mac 00:11:22:33:44:55
ip link set eth0 vf 0 ns testns
ip netns exec testns ip link set dev eth0 name vf0 up

# Create bridge
ip link add name br0 type bridge
ip link set dev br0 up
# Disable learning/filtering to isolate forwarding cost
bridge link set dev br0 age_time 0   # disable ageing (optional)
bridge vlan del dev br0 vid 0        # ensure VLAN filtering off
# Add VF and veth to bridge
ip link add veth0 type veth peer name veth1
ip link set veth0 netns testns
ip link set veth1 master br0 up
ip netns exec testns ip link set veth0 up
ip netns exec testns ip link set veth1 up
# Optional: turn off STP
bridge link set dev br0 stp_state 0

Verify: bridge link show should list both ports with state forwarding.

Results

Observations

• The bridge adds roughly 2–3 µs of fixed processing overhead per packet (MAC lookup, possible hash table update).
• CPU usage scales inversely with packet size because the per‑packet cost dominates; larger packets amortize the cost over more payload.
• With bridge ageing disabled, the MAC table does not grow; enabling ageing adds a small periodic cost (~0.2 % CPU per 10k entries).

Operational Caveats

• MAC learning limits – default hash table size 4096 entries; exceeding it causes floods and CPU spikes.
• VLAN filtering – when enabled, each packet incurs an extra VLAN header check; can drop performance ~5‑10 % for tagged traffic.
• ARP/ND proxy – bridge does not answer ARP; you must rely on the host or use proxy_arp.
• No built‑in QoS – traffic shaping must be done via tc on the bridge or member devices.
• Interaction with offloads – GRO/GSO can be enabled on bridge ports, but L2 forwarding disables hardware checksum offload for packets that are modified (e.g., VLAN push/pop). Verify with ethtool -k br0.

Open vSwitch (OVS)

Configuration

# Install OVS (userspace + kernel module)
apt-get install -y openvswitch-switch
systemctl start openvswitch-switch

# Create bridge
ovs-vsctl add-br ovsbr0
# Set datapath type to system (kernel) for fast path
ovs-vsctl set bridge ovsbr0 datapath_type=system

# Disable STP and other unnecessary features
ovs-vsctl set bridge ovsbr0 stp_enable=false
ovs-vsctl set bridge ovsbr0 other_config:disable-in-band=true

# Add VF as netdev port
ip link set eth0 vf 0 mac 00:11:22:33:44:55
ip link set eth0 vf 0 ns testns
ip netns exec testns ip link set dev eth0 name vf0 up
ovs-vsctl add-port ovsbr0 vf0 -- set Interface vf0 type=internal

# Create veth pair for test traffic
ip link add veth0 type veth peer name veth1
ip link set veth0 netns testns
ovs-vsctl add-port ovsbr0 veth1
ip netns exec testns ip link set veth0 up
ip link set veth1 up

# Install a simple flow: all packets go to veth1
ovs-ofctl add-flow ovsbr0 "table=0, priority=0, actions=output:veth1"
# Verify flow table
ovs-ofctl dump-flows ovsbr0

Important: Ensure n_handler_threads matches the number of cores you intend to poll (ovs-vsctl set Open_vSwitch . other_config:n_handler_threads=2).

Results

Observations

• OVS kernel datapath adds ~0.8 µs more per packet than the Linux bridge for 64 B, mainly due to the extra flow lookup (hash in the kernel datapath) and the upcall miss path (though our flow hits, there is still a small per‑packet cost for the struct sw_flow_key construction).
• CPU overhead is slightly higher than the bridge because each packet traverses the vport_netdev receive path and the dp_packet metadata handling.
• With n_handler_threads=2 we see ~13 % CPU on each handler core; total ~26 % as reported.

Operational Caveats

• Flow table size – the kernel datapath uses a fixed‑size hash (default 65536 entries). Exceeding it causes fallback to userspace (ovs-vswitchd) and massive latency spikes.
• Miss handling – any packet not matching a flow triggers an upcall; under heavy miss rates the userspace daemon can become a bottleneck (>10 µs per packet).
• Feature interaction – enabling features like geneve, vxlan, or mirror adds extra encapsulation/decap steps, each costing ~1‑2 µs.
• Offloads – OVS can offload flow actions to NIC hardware via TC flower (tc filter add ... action mirred egress redirect dev eth0) but only if the NIC supports the specific offload (e.g., Mellanox hardware offload for L2 forwarding). Without offload, all processing stays in software.
• Memory – each flow consumes ~1 KB; large flow tables can pressure the slab allocator.

Macvlan

Configuration

# Create macvlan device in bridge mode (appears as separate MAC on same lower dev)
ip link add link eth0 macvlan0 type macvlan mode bridge
# Move it into test namespace
ip link set macvlan0 netns testns
ip netns exec testns ip link set dev macvlan0 up

# Create veth pair for test traffic (one end in ns, other in host for pktgen)
ip link add veth0 type veth peer name veth1
ip link set veth0 netns testns
ip netns exec testns ip link set veth0 up
ip link set veth1 up

# Inside namespace, attach veth0 to macvlan0 via a simple bridge (optional)
# For pure macvlan we just use the macvlan as the endpoint:
ip netns exec testns ip link set dev veth0 master macvlan0
# Alternatively, you can use veth0 directly as the traffic endpoint:
# ip netns exec testns ip addr add 192.168.100.2/24 dev veth0
# ip netns exec testns ip link set dev veth0 up

Note: In VEPA mode (mode vepa) the macvlan forwards all packets to the external switch; hairpin mode (mode passthu) allows local switching. For a fair L2 comparison we use bridge mode, which behaves like a simple bump‑in‑the‑wire with no learning.

Results

Observations

• Macvlan adds the least overhead: essentially only the cost of the macvlan driver’s packet steering and the veth pair.
• Latency is lowest because packets bypass any software learning or flow‑lookup tables; they are forwarded directly by the NIC (or its software emulation) to the destination MAC.
• CPU usage is minimal and remains fairly constant across packet sizes.

Operational Caveats

• No L2 learning – macvlan does not maintain a forwarding table; traffic between two macvlan interfaces on the same lower device must go out to the external switch and back (hairpin mode required for local switching).
• Limited multicast/broadcast handling – depending on the mode (vepa, bridge, private), broadcast traffic may be dropped or flooded incorrectly.
• No VLAN filtering – VLAN tags are passed through unchanged