Reading FDB aging like an outage timeline

Introduction to Short Bridge FDB and Packet‑Capture Artifacts

The Linux bridge maintains a Forwarding Database (FDB) that maps MAC addresses to bridge ports. Entries are static (admin‑added) or dynamic (learned from the source MAC of incoming frames). For each ingress frame the bridge looks up the destination MAC: if found, the frame is forwarded only to the associated port; otherwise it is flooded to all ports except the ingress port.

The FDB is a short‑lived cache. Entries are removed when they become stale (no traffic seen for the aging interval) or when the FDB table fills and the oldest entry is evicted. The aging timer is configurable per bridge (default 300 s).

When workloads observe intermittent packet loss, the symptom often appears as random drops. In a bridged environment the loss is frequently caused by unknown‑unicast flooding bursts: the bridge floods a frame because its destination MAC is missing from the FDB, consuming bandwidth on ports that do not need the traffic and causing queue drops on those ports. Capturing traffic on a port that should not see the flow reveals the burst pattern, while inspecting the FDB shows why the MAC was missing (stale entry, rapid relearn, or aging‑timer expiry).

Understanding Stale Entries and Their Impact

Definition and Causes

A stale FDB entry is a dynamic mapping that has not been refreshed by a source‑MAC learning event within the bridge’s aging interval. Common causes:

• VM/container moves to another bridge port (or leaves) without generating traffic from its MAC for the duration of the aging timer.
• Traffic asymmetry: only one direction of a flow is seen, so the reverse‑direction MAC is never refreshed.
• Bursty workloads followed by long idle periods (e.g., batch jobs, backups).

When an entry becomes stale, the bridge treats any frame destined for that MAC as unknown‑unicast and floods it until the MAC is relearned (by seeing a frame with that MAC as source).

Effects on Network Performance

• Flooding overhead – each unknown‑unicast frame is sent out all bridge ports, increasing load on unrelated links.
• Queue buildup & drops – ports not part of the flow may experience transient congestion, leading to packet loss for unrelated workloads.
• Latency spikes – flooded frames incur store‑and‑forward delay on each egress port.
• Apparent random loss – flooding occurs only when a stale entry times out, making the loss appear periodic and uncorrelated with application logs.

Identifying Stale Entries with Packet Captures

Capture on a quiet port (a bridge port that carries no expected traffic for the flow) and look for bursts of frames whose destination MAC equals the MAC of the silent host.

# Capture on veth10 (quiet port) for 10 s, show MAC headers
sudo tcpdump -i veth10 -e -nn -s 0 -c 200 'ether[6:2] = 0x0800'

Sample output (truncated):

12:34:56.789012 00:11:22:33:44:55 > 00:aa:bb:cc:dd:ee, ethertype IPv4 (0x0800), length 98: ...
12:34:56.789150 00:11:22:33:44:55 > 00:aa:bb:cc:dd:ee, ethertype IPv4 (0x0800), length 98: ...
12:34:56.789300 00:11:22:33:44:55 > 00:aa:bb:cc:dd:ee, ethertype IPv4 (0x0800), length 98: ...

Here 00:aa:bb:cc:dd:ee is the MAC of a host not attached to veth10. Seeing multiple frames with that destination in a short window indicates the bridge flooded an unknown‑unicast frame for that MAC.

Correlate with the FDB:

sudo bridge fdb show dev br0 | grep 00:aa:bb:cc:dd:ee

If the entry is missing, marked self, or shows an old timestamp, the entry is stale.

Rapid Relearns and Their Role in Periodic Unknown‑Unicast Bursts

What Are Rapid Relearns?

A rapid relearn occurs when a host’s MAC address flaps between bridge ports faster than the aging timer can expire the old entry. Each time the MAC appears on a new port, the bridge overwrites the FDB entry with the new port. If the host moves back and forth (e.g., mis‑configured bonding, container live‑migration loops, VMware vMotion misbehavior), the entry is constantly updated, causing the bridge to flush and relearn the MAC on each change.

When the MAC flaps, there is a brief window after each move where the bridge has no entry for that MAC on the previous port (the old entry still exists but points to the wrong port). Frames destined for the MAC that arrive on the old port are flooded until the new entry is learned. If the flap frequency matches the aging timer or is a sub‑multiple, the flooding appears as a periodic burst.

Relationship to Stale Entries

Rapid relearns generate temporary stale entries on the port the host just left. The bridge considers the entry stale only after the aging timer expires without seeing a source MAC on that port. If the host returns before the timer expires, the entry is simply updated (not stale). However, if the host leaves and does not return before the timer expires, the entry becomes stale and will cause flooding for any traffic destined to that MAC that arrives on the old port after the timer expires. In high‑frequency flapping, the timer may never expire, but constant overwriting can still cause micro‑bursts of flooding each time the MAC changes direction.

Detecting Rapid Relearns

Monitor FDB changes in real time:

sudo bridge monitor dev br0

Sample output during a flap (MAC 00:11:22:33:44:55 moving between veth5 and veth6 every 2 s):

00:11:22:33:44:55 dev veth5 self permanent
00:11:22:33:44:55 dev veth6 self
00:11:22:33:44:55 dev veth5 self
00:11:22:33:44:55 dev veth6 self
...

Each line indicates the bridge relearned the MAC to a new port. To count relearns over a period:

sudo bridge monitor dev br0 2>/dev/null | grep -c "00:11:22:33:44:55"

If the count spikes (e.g., >10 relearns in 5 s) while traffic counters on the ports show no corresponding increase, suspect rapid relearns.

Aging Timers and Their Contribution to Unknown‑Unicast Bursts

Functionality and Configuration

The bridge aging timer defines how long a dynamic FDB entry is retained without seeing a source‑MAC refresh. When the timer expires, the entry is flushed. The timer is set per bridge:

# Show current aging time (seconds)
sudo bridge fdb show dev br0 | grep ageing_time
# Output: ageing_time 300

# Set a new aging time (e.g., 60 s)
sudo bridge fdb set dev br0 ageing_time 60

A short aging timer causes entries to expire quickly, increasing the chance that a host silent longer than the timer will be treated as unknown‑unicast. A long timer reduces false flooding but can cause blackholing if a host moves and the old entry persists longer than the new traffic arrives.

Interaction with Stale Entries and Rapid Learns

• Stale entry creation – occurs when a host stops sending traffic for longer than the aging interval.
• Rapid relearn mitigation – if the aging timer is shorter than the flap period, the old entry may expire before the host returns, turning a flap into a stale entry and causing a flood burst when the host finally sends a frame.
• Burst periodicity – the observed burst interval often approximates the aging timer (or a multiple thereof) because each expiration triggers a flood until the MAC is relearned.

Example: Adjusting the Aging Timer

Reduce flooding for a host known to be silent up to 90 s:

# Current timer
sudo bridge fdb show dev br0 | grep ageing_time
# ageing_time 300

# Reduce to 90 s to match expected silence
sudo bridge fdb set dev br0 ageing_time 90

# Verify
sudo bridge fdb show dev br0 | grep ageing_time
# ageing_time 90

If the environment experiences rapid MAC flapping (e.g., VM live‑migration loops), increase the timer to avoid premature expiry:

sudo bridge fdb set dev br0 ageing_time 600   # 10 min

Troubleshooting Periodic Unknown‑Unicast Bursts

Using Packet‑Capture Artifacts to Identify Burst Patterns

1. Select a monitoring port that should not see the flow (e.g., a VLAN interface used only for management).
2. Capture with microsecond resolution and limit to Ethernet type IPv4/IPv6 to reduce noise.
3. Look for periodic spikes in the number of packets with a specific destination MAC.

# Capture 30 s, output to file for later analysis
sudo tcpdump -i mgmt0 -e -nn -s 0 -w /tmp/unknown_ucast.pcap 'ether[6:2] = 0x0800'

After capture, use tshark to count per‑destination MAC:

tshark -r /tmp/unknown_ucast.pcap -Y eth.dst -T fields -e eth.dst | sort | uniq -c | sort -nr | head -10

A MAC that appears with a high count in bursts (e.g., 500 packets every 20 s) while no traffic is expected indicates unknown‑unicast flooding.

Analyzing the Short Bridge FDB to Detect Stale Entries and Rapid Relearns

Combine FDB snapshots with timestamps:

# Take a snapshot every second for 20 s
for i in {1..20}; do
    date +"%T.%3N" >> /tmp/fdb_ts.log
    sudo bridge fdb show dev br0 >> /tmp/fdb_ts.log
    sleep 1
done

Parse the log: if a MAC’s associated port changes frequently or disappears for a period longer than the aging timer, you have a stale entry or rapid relearn.

CLI Commands for Troubleshooting Aging Timer Issues

• Show current timer:

  sudo bridge fdb show dev br0 | grep ageing_time

• Show entry age (kernel ≥ 4.9; bridge fdb show includes a timestamp in jiffies):

  sudo bridge fdb show dev br0 | awk '{print $1,$2,$3,$4,$5}'

  The third column is the age in seconds (if supported).

• Flush all dynamic entries to force relearn (use with caution):

  sudo bridge fdb flush dev br0

• Monitor for flush events:

  sudo bridge monitor dev br0 | grep flush

Scaling Limitations and Considerations

Impact of Network Size on Short Bridge FDB and Aging Timers

• The FDB is a fixed‑size hash table (default 8192 entries on many kernels). In large Layer‑2 domains (hundreds of thousands of MACs), the table can overflow, causing eviction of the oldest entry regardless of its age. This mimics aging‑timer behavior and can produce bursts even when timers are long.
• Aging timers are global per bridge; tuning them for a subset of hosts affects all. In very large fabrics, a single timer value may be suboptimal for both stationary and mobile workloads.

Scaling the Short Bridge FDB and Aging Timers for Large Networks

• Increase FDB size via sysctl (if supported):

  sudo sysctl -w net.bridge.bridge-max-addr=65536

  (Verify the exact parameter with sysctl -a | grep max-addr.)

• Use multiple bridge instances (e.g., one per rack or VLAN) to limit the MAC domain size per FDB.

• For highly dynamic environments, consider turning off learning (bridge link set dev br0 learning off) and rely on static FDB entries or external SDN controllers for forwarding decisions.

Limitations of Packet‑Capture Artifacts in High‑Volume Networks

• Capturing on a high‑throughput port can drop packets if the capture tool cannot keep up, skewing burst visibility.
• Storage consumption grows linearly; long captures may fill disks.
• In environments with hardware offload (e.g., SmartNICs, eSwitch), the software bridge may not see all packets; captures on the host NIC may miss traffic that is switched in hardware.
• Mitigation: use ring‑buffer capture (-C 100 -W 5) to rotate files, or capture on a mirror port (tc qdisc add dev eth0 handle ffff: ingress + tc filter add dev eth0 parent ffff: protocol ip u32 match u8 0 0 action mirred egress mirror dev mon0).

Mitigation Strategies for Periodic Unknown‑Unicast Bursts

Adjusting Aging Timers for Optimal Performance

• Base the timer on the maximum expected silence of the majority of workloads, measured via flow‑monitoring (e.g., using netstat, ss, or external telemetry).
• Apply the timer per bridge; if different workloads have vastly different silence periods, consider segmenting them into separate bridges or VLANs.

Reducing Rapid Relearns

• Identify and correct the source of MAC flapping (mis‑configured bonding, faulty live‑migration scripts, NIC driver issues).
• If flapping is unavoidable, increase the aging timer so the old entry does not expire prematurely, or disable learning on the bridge and install static FDB entries for the mobile hosts.

Controlling FDB Size

• Monitor FDB utilization (bridge fdb show dev br0 | wc -l).
• If utilization approaches the limit, increase net.bridge.bridge-max-addr or split the bridge.

Using Static FDB Entries for Critical Hosts

• For hosts that must never be flooded (e.g., gateway routers, load balancers), add static entries:

  sudo bridge fdb add 00:11:22:33:44:55 dev br0 veth12 master

• Static entries are not subject to aging or relearn, eliminating unknown‑unicast bursts for those MACs.

Verifying Mitigation

After applying changes, repeat the packet‑capture procedure on a quiet port. The burst count for the previously problematic MAC should drop to zero or to baseline levels. Confirm with FDB queries that the entry is present, correctly aged, and not flapping.

By correlating packet‑capture artifacts with FDB state, administrators can pinpoint whether stale entries, rapid relearns, or aging‑timer settings are causing periodic unknown‑unicast bursts, and then apply targeted tuning to eliminate the apparent random packet loss experienced by workloads.