The bridge MTU was fixed but the path still broke

Introduction to MTU and Fragmentation

The Maximum Transmission Unit (MTU) defines the largest payload, in bytes, that a layer‑2 frame can carry without fragmentation. On Ethernet the default is 1500 bytes. When a packet exceeds the MTU of any egress interface, the IP layer fragments it, adding per‑fragment headers, requiring reassembly, and potentially causing loss if any fragment is dropped. In high‑throughput or low‑latency workloads (RDMA, VXLAN, GRE, container‑to‑container traffic) unnecessary fragmentation degrades jitter and increases CPU overhead.

Fragmentation appears when:

An application or tunnel encapsulates payload that pushes the total size above the smallest MTU in the path.
A virtual interface (tap, veth, macvlan) is created with a default MTU of 1500 while the underlying physical or bridge MTU is larger.
Mis‑configured QoS or offload features (GRO/LRO) inadvertently reduce the effective MTU seen by the stack.
Container‑lab tools attach veth pairs to bridges without propagating MTU changes, leaving a hidden choke point.

The Problem: Raising Bridge MTU Had No Effect

In a Containerlab topology we observed TCP retransmits and “frag needed” ICMP messages when sending 1500‑byte payloads over a VXLAN tunnel terminating on a bridge. Increasing the bridge MTU:

sudo ip link set dev br0 mtu 9000

did not resolve the issue; ping -M do -s 1472 <target> still returned “Frag needed”. The bridge MTU change had no effect because another link in the path limited the MTU.

Lab setup

Two Containerlab nodes (clab-rt1, clab-rt2) running FRR.
Linux bridge br0 on the host, to which the nodes’ veth interfaces are attached.
VXLAN interfaces (vxlan10) on each FRR instance encapsulating traffic over the bridge.

Symptoms:

tcpdump -i br0 -nn showed IP packets with the MF (More Fragments) flag.
ip -s link show br0 reported RX/TX errors incrementing with each large packet.
Disabling GRO/LRO on br0 did not eliminate fragmentation.

Identifying the Real Choke Point

A veth pair is always created with an MTU of 1500 unless explicitly set otherwise. When a veth is attached to a bridge, the bridge does not inherit or enforce the veth’s MTU; each interface maintains its own MTU. If any interface in the forwarding path has a lower MTU, the IP layer fragments before the packet reaches the bridge.

Path for a packet from clab-rt1’s VXLAN interface to the physical NIC

vxlan10 (FRR) – MTU inherited from underlying socket (default 1500).
veth-clab-rt1 – MTU 1500 (default).
br0 – MTU 9000 (after our change).
veth-clab-rt2 – MTU 1500 (default).
vxlan10 (FRR on rt2) – MTU 1500.
Physical NIC (e.g., eth0) – MTU 1500.

Even though br0 supports jumbo frames, the veth pairs and VXLAN interfaces remain at 1500, forcing fragmentation at step 2 or step 4 before the bridge ever sees the frame.

Using CLI Tools to Inspect Interface Settings

ip link show lists the MTU of every interface.
ip -d link show <iface> reveals detailed attributes, including LOWER_UP state.
tc -s qdisc show dev <iface> can expose queueing disciplines that might artificially limit MTU (rare, but possible with netem).

CLI Examples for Troubleshooting MTU Issues

Inspect Interface MTUs

# List all interfaces and their MTU
ip -brief link show

Sample output:

lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP> mtu 65536
eth0             UP             00:11:22:33:44:55 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
br0              UP             00:aa:bb:cc:dd:ee <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000
veth-clab-rt1    UP             00:11:22:33:44:66 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
veth-clab-rt2    UP             00:11:22:33:44:77 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
vxlan10          UP             00:00:00:00:00:00 <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500

The veth and VXLAN MTUs remain 1500.

Capture Fragmentation with tcpdump

sudo tcpdump -i br0 -nn -s 0 -v 'ip[6] & 0x3f != 0'

Example output:

12:34:56.789012 IP (tos 0x0, ttl 64, id 12345, offset 0, flags [MF], proto TCP (6), length 1500)
    10.0.0.1.47890 > 10.0.0.2.80: Flags [P.], seq 1:1461, ack 1, win 256, length 1460
12:34:56.789123 IP (tos 0x0, ttl 64, id 12345, offset 1460, flags [MF], proto TCP (6), length 1500)
    10.0.0.1.47890 > 10.0.0.2.80: Flags [P.], seq 1461:2921, ack 1, win 256, length 1460
12:34:56.789234 IP (tos 0x0, ttl 64, id 12345, offset 2920, flags [DF], proto TCP (6), length 80)
    10.0.0.1.47890 > 10.0.0.2.80: Flags [P.], seq 2921:3001, ack 1, win 256, length 80

The [MF] flag indicates more fragments follow; the final fragment lacks [MF] and has [DF] set, confirming the sender is fragmenting because the next hop’s MTU is too small.

Determine Path MTU with tracepath

tracepath -n 10.0.0.2

Sample output:

 1?: [LOCALHOST]                      pmtu 1500
 1:  10.0.0.1                         0.123ms pmtu 1500
 2:  10.0.0.2                         0.456ms reached
 Resume: pmtu 1500

Even with a jumbo‑capable bridge, tracepath reports 1500 because the veth limits it.

Scaling Limitations and Considerations

Larger MTUs reduce per‑packet header overhead, improving throughput for bulk transfers. However, they increase head‑of‑line blocking risk in software switches and can expose NIC driver bugs with jumbo frames. In container‑heavy environments, many veth pairs are created; setting each to a non‑default MTU adds operational complexity and can cause mismatches if automation does not propagate the change consistently.

Common Pitfalls

Assuming that changing a bridge MTU propagates to attached veth/tap interfaces—it does not.
Forgetting that VXLAN adds 50 bytes of outer header (8 bytes UDP + 20 bytes IP + 20 bytes outer Ethernet); the inner MTU must be reduced accordingly if the outer path cannot accommodate the extra bytes.
Relying on the default MTU of 1500 for all interfaces in automation scripts; always explicitly set MTU where required.

Best Practices

Define a canonical MTU (e.g., 9000 for jumbo‑capable fabrics) and apply it to every layer‑2 object: physical NICs, bridges, bonds, veth/tap, and any overlay interfaces.
Validate with ip link show after any interface creation or modification.
Automate checks in CI pipelines: a simple script that ip -brief link show and asserts all MTUs equal the desired value.
Document overlay overhead: for VXLAN set the underlying MTU to at least payload MTU + 50; for GRE add 24 bytes; for IPsec ESP add up to 50 bytes depending on encryption mode.

Advanced Troubleshooting Techniques

ethtool – Ring Buffers and Coalesce

Ring buffers do not affect MTU, but they influence whether fragmented packets are dropped under load.

# Increase RX ring to reduce drops during bursty fragmented traffic
sudo ethtool -G eth0 rx 4096 tx 4096
# Reduce coalesce latency to favor smaller packets
sudo ethtool -C eth0 rx-usecs 20 tx-usecs 20

These tweaks mitigate symptoms while the root cause is addressed.

sysctl – Fragment Reassembly Tuning

# Allow the kernel to accept larger fragmented packets (default is 65535)
sudo sysctl -w net.ipv4.ip_frag_high_thresh=262144
sudo sysctl -w net.ipv4.ip_frag_low_thresh=196608
# Reduce the timeout for fragment reassembly (helps detect loss faster)
sudo sysctl -w net.ipv4.ipfrag_time=10

Again, these are mitigations; the proper fix is to eliminate the need for fragmentation.

Automation One‑Liner

Find any interface with MTU < 9000 on a host:

ip -brief link show | awk '$3 < 9000 {print $1, $3}'

Output:

veth-clab-rt1 1500
veth-clab-rt2 1500
vxlan10 1500

This immediately highlights the choke points.

Locating the Real Choke Point: Step‑by‑Step Guide

Step 1: Inspect All Interfaces in the Traffic Path

# On the host, list every veth attached to the bridge
bridge link show | grep -E 'veth|br0' | awk '{print $2, $5, $9}'
# On each container, check internal interfaces
docker exec clab-rt1 ip -brief link show
docker exec clab-rt2 ip -brief link show

Collect MTU values for:

Physical NIC (eth0)
Bridge (br0)
Each veth pair (veth-<container>)
VXLAN interfaces (vxlan10)
Any other overlays (GRE, VLAN, etc.)

Step 2: Identify and Adjust the Smallest MTU

Determine the minimum MTU observed. If it is 1500 and you need jumbo frames, raise all interfaces to the target value:

# Host side
sudo ip link set dev eth0 mtu 9000
sudo ip link set dev br0 mtu 9000
sudo ip link set dev veth-clab-rt1 mtu 9000
sudo ip link set dev veth-clab-rt2 mtu 9000
# Inside containers (if needed)
docker exec clab-rt1 ip link set dev eth0 mtu 9000
docker exec clab-rt2 ip link set dev eth0 mtu 9000
# Overlay interfaces (adjust inner MTU to accommodate overhead)
docker exec clab-rt1 ip link set dev vxlan10 mtu 8950   # 9000 - 50 bytes VXLAN overhead
docker exec clab-rt2 ip link set dev vxlan10 mtu 8950

After these changes, re‑run the ping test:

ping -M do -s 8950 <target>   # tests a 9000‑byte IP packet

You should now receive replies without “Frag needed”, confirming the choke point has been removed.