Trace an Ethernet Frame from a Tap‑backed Guest NIC to a Destination Namespace

Ethernet Frame Structure

An Ethernet frame is a 14‑byte header (dst MAC 6 B, src MAC 6 B, EtherType/len 2 B), optional 802.1Q tag (4 B), payload (46‑1500 B for standard MTU, up to 9 KB for jumbo), and a 4‑byte FCS. Once the NIC’s DMA ring delivers the frame to the kernel, it becomes a struct sk_buff (skb).

Why Trace the Frame?

Tracing lets you pinpoint latency, drops, or mis‑ordering. In a tap‑backed guest‑to‑namespace flow the frame crosses several kernel boundaries (tap → bridge → veth pair → namespace stack). Each hop adds queuing, lookups, and possible offload effects. Without per‑hop visibility you can only infer problems from end‑to‑end metrics, which often mislead operators.

Guest NIC Transmission

Inside the guest a virtio‑net (or similar) device presents a virtual NIC. When the guest’s stack hands an skb to the device:

1. Guest writes to the virtio transmit queue.
2. Vhost‑net kicks the host’s tap0 rx handler (tap_open → tap_rx).
3. The tap driver allocates an skb, copies the payload, sets skb->dev = tap0, and calls netif_rx().

At this point the frame is untagged, retains the guest’s source MAC, and carries the original EtherType.

Host Bridge Reception

The tap device (tap0) is enslaved to a Linux bridge (br0). When netif_rx() delivers the skb to tap0, the bridge’s br_handle_frame() runs via the netdev’s rx_handler. The bridge performs:

• Ingress timestamping (if SO_TIMESTAMPING enabled).
• VLAN header extraction (if VLAN filtering active).
• Delivery to forwarding logic (br_handle_frame_finish()).

The bridge does not do a routing lookup; it consults its forwarding database (FDB) to decide which local port(s) to forward the frame to.

FDB Lookup Process

The bridge maintains an FDB (struct net_bridge_fdb_entry) protected by RCU. On receipt of a frame:

1. br_fdb_find_addr(br, skb->dst) is called under rcu_read_lock().
2. If a matching entry exists and its port ≠ incoming port, the frame is marked for forwarding to that port (br_forward()).
3. If no entry is found (unknown unicast) or the destination is broadcast/multicast, the frame is flooded to all ports except the incoming one (br_flood()).
4. The FDB entry is updated (or created) for the source MAC with the incoming port and a timestamp; aging is handled by br_fdb_cleanup() (default 300 s).

Lookup is O(1) hash table access; under heavy load the RCU read‑side critical section can become a contention point when many CPUs update the FDB (e.g., VM migration or MAC flapping).

Qdisc Touchpoints and Traffic Shaping

Each bridge port has its own egress qdisc (default pfifo_fast). The flow after the FDB decision:

1. The skb is handed to the outgoing port’s dev_queue_xmit().
2. The outgoing port’s qdisc (br_port->dev->qdisc) is invoked:
   • Classful schedulers (htb, cbq) classify the skb into a band based on tc filter rules or skb priority.
   • Simple FIFO (pfifo_fast) enqueues the skb directly.
3. The qdisc may drop the packet if its queue length exceeds the configured limit (txqueuelen or qlen).
4. When the NIC driver is ready, the qdisc’s dequeue() callback pulls skbs and hands them to the driver’s hard_start_xmit().

Important: the bridge has no ingress qdisc; any ingress shaping must be applied on the tap device or on the veth peer before the bridge sees the frame.

If the outgoing port uses a classful qdisc (e.g., htb with rate ceilings), the skb acquires a tc_index that determines its band. The scheduler updates the band’s byte counter; if the band exceeds its ceiling the skb may be delayed (rate‑limited) or marked with TC_PRIO_MAX for later dropping. The qdisc’s internal lock (spin_lock_bh(&qdisc->lock)) protects the queue; under burst traffic this lock can become a hotspot, increasing latency and creating back‑pressure onto the bridge’s ingress path.

When the skb reaches the driver, it is handed to the NIC’s transmit ring. If the ring is full, the driver returns NETDEV_TX_BUSY, the qdisc retains the skb, and the bridge sees increased queue depth.

Namespace Routing and Delivery

Assume the bridge port that won the FDB lookup is a veth pair (veth0 ↔ veth1). veth0 is enslaved to br0; veth1 lives inside the destination netns (ns-dst). After egress qdisc on veth0, the frame is transmitted to its peer veth1 via the veth driver’s veth_xmit():

1. Allocate a new skb (or reuse the original if skb_shared_ok).
2. Copy the payload (zero‑copy when possible).
3. Set skb->dev = veth1.
4. Call netif_rx() on veth1 inside the target namespace (the veth driver switched netns at creation time via dev_change_net_namespace()).

Thus the frame appears as an incoming packet on veth1 in ns-dst. The namespace’s networking stack processes it exactly as if it had arrived on a physical NIC:

• Ingress qdisc on veth1 (if any) runs.
• The skb is passed to netif_receive_skb() → ip_rcv() (if EtherType is IPv4/IPv6) or handed to a userspace socket via AF_PACKET/TPACKET_v3.
• If the destination is a local address (another veth or loopback), the IP layer performs a routing lookup (fib_lookup()) using the namespace’s own FIB rules and tables.
• The frame is then delivered to the appropriate socket or forwarded out another interface.

If the destination is a service listening on a socket bound to veth1’s IP, the frame is ultimately copied to user space via recvmsg(). If the destination is another veth leading to yet another namespace or a physical NIC, the frame repeats the bridge/veth cycle in the outbound direction.

Troubleshooting Ethernet Frame Issues

Common Symptoms and Likely Locations

Debugging Tools and Techniques

• Tap‑side: tcpdump -i tap0 -w guest.pcap captures exactly what the guest sent.
• Bridge ingress: tcpdump -i br0 -e -w br_ingress.pcap (-e shows MACs).
• Bridge egress per port: tcpdump -i veth0 -e -w br_egress.pcap.
• Namespace ingress: nsenter -t $(cat /var/run/ns-dst.pid) -n tcpdump -i veth1 -w ns_ingress.pcap.
• eBPF/XDP tracing:

  bpftrace -e 'tracepoint:net:netif_receive_skb /args->skb->dev->name == "veth1"/ { @[kstack] = count(); }'

• Statistics:

  bridge fdb show
  bridge link show
  tc -s qdisc show dev <if>
  ethtool -S <if>          # ring drops
  cat /proc/net/dev        # rx/tx drops per iface

Example Troubleshooting Scenarios

1. Guest → Bridge, no reply

   • Capture on tap0: see ARP request.
   • Capture on br0: ARP request appears, but no ARP reply.
   • Check bridge FDB: bridge fdb show br0 shows only the guest’s MAC on veth0.
   • The destination namespace’s veth1 has no IP assigned → ARP never answered.
   • Fix: ip addr add 10.0.0.2/24 dev veth1 inside ns-dst.

2. High latency under TCP burst

   • tc -s qdisc show dev veth0 shows growing backlog and drops.
   • ethtool -i veth0 reveals tx-ring size 256, but ethtool -g veth0 shows current usage near 200.
   • Increase ring: ethtool -G veth0 tx 1024 or switch to fq qdisc:

     tc qdisc replace dev veth0 root fq

3. VLAN‑tagged frames dropped

   • Guest sends VLAN 100 tagged frame.
   • tcpdump -i tap0 -e shows 802.1Q tag.
   • Bridge port veth0 has vlan filtering 1 but no VLAN 100 configured → frame dropped silently.
   • Fix:

     bridge vlan add dev veth0 vid 100 pvid untagged

Code and CLI Examples

Capturing Ethernet Frames with tcpdump

# Guest‑side tap
tcpdump -i tap0 -w /tmp/guest_tap.pcap -s 0 -nn

# Bridge ingress (all ports)
tcpdump -i br0 -w /tmp/br_ingress.pcap -s 0 -nn

# Specific egress port (veth0)
tcpdump -i veth0 -w /tmp/veth0_egress.pcap -s 0 -nn

# Namespace ingress (veth1 inside ns-dst)
nsenter -t $(cat /var/run/ns-dst.pid) -n \
    tcpdump -i veth1 -w /tmp/ns_ingress.pcap -s 0 -nn

These commands give you a complete view of the frame at each hop: guest tap, bridge ingress, bridge egress via the veth port, and finally inside the destination namespace. Adjust interface names and paths as needed for your setup.