Hidden offloads that lie to your packet capture

Introduction to Network Offloading and Capture

Overview of veth Pairs and Bridges

A veth pair acts like a virtual Ethernet cable: packets sent on one end appear on the peer end, preserving the original skb unless the kernel clones or modifies it. Each end can be placed in a different network namespace, making veth the basic building block for container‑to‑container or container‑to‑host communication.

A Linux bridge behaves like an Ethernet switch: it learns MAC addresses, forwards frames based on destination MAC, and floods unknown destinations. When a veth endpoint is enslaved to a bridge, the bridge treats it as a physical port; the veth driver delivers the skb to the bridge’s ndo_start_xmit hook, which then invokes the bridge’s forwarding logic.

Both veth and bridge are pure software datapaths, yet they expose the same offload capabilities as a real NIC: checksum offload, GRO, GSO, TSO, and UFO. These features are implemented in the veth driver as flags that defer work until later in the transmit path.

Importance of Checksum and GRO Offload

Checksum offload allows the NIC (or veth) to transmit a packet with an incomplete L3/L4 checksum; the actual calculation is performed either by hardware (if present) or by the kernel just before the packet leaves the host stack. Capturing on the transmit side may show a zero or incorrect checksum, causing tools like tcpdump or Wireshark to flag the packet as corrupt, even though the kernel will compute the correct checksum before final transmission.

GRO coalesces multiple inbound packets that share the same 5‑tuple into a single larger‑than‑MTU skb, reducing per‑packet processing overhead. The resulting GRO segment carries concatenated data; the original packets are no longer visible on the receive path. Capturing after GRO shows fewer, larger packets; capturing before GRO shows the original segments.

In an emulated environment built from veth pairs and bridges, the same offload logic runs in software. If you are unaware that the veth device has checksum or GRO enabled, a capture can appear “broken” (bad checksums, unexpected packet sizes) even though the datapath is functioning exactly as the kernel designed.

Setting Up the Emulated Network Environment

Creating veth Pairs

# Create two veth pairs: veth0<->veth1 and veth2<->veth3
ip link add veth0 type veth peer name veth1
ip link add veth2 type veth peer name veth3

# Move one end of each pair into its own namespace
ip netns add left
ip netns add right
ip link set veth1 netns left
ip link set veth3 netns right

# Bring up the interfaces
ip link set veth0 up
ip link set veth2 up
ip netns exec left ip link set veth1 up
ip netns exec right ip link set veth3 up

At this point veth0 and veth2 reside in the root namespace, while veth1 and veth3 are isolated in left and right.

Configuring Bridges

# Create bridge br0
ip link add name br0 type bridge
ip link set br0 up

# Attach the veth ends to the bridge
ip link set veth0 master br0
ip link set veth2 master br0

# Verify
bridge link show

The bridge now has two ports: veth0 and veth2. Frames arriving on either port are flooded to the other unless the destination MAC is known.

Building a Custom Node Image

# Dockerfile for custom node image
FROM scratch
COPY busybox /bin/
COPY iproute2/sbin/ip /sbin/ip
COPY ethtool/sbin/ethtool /sbin/ethtool
COPY tcpdump/usr/sbin/tcpdump /usr/sbin/tcpdump
COPY iperf3/usr/bin/iperf3 /usr/bin/iperf3
# Minimal dev nodes
RUN mknod /dev/null c 1 3 && mknod /dev/zero c 1 5 && mknod /dev/random c 1 8 && mknod /dev/urandom c 1 9
ENTRYPOINT ["/bin/sh"]

docker build -t node-img .
docker create --name node node-img
docker export node | sudo tar -C /var/lib/netns/node-img -x
ip netns add node-img
# Bind‑mount the filesystem into the namespace (requires nsenter or similar)

The image contains no NIC drivers; all networking is provided by the veth/bridge topology we created earlier. This guarantees that any offload observed is purely software‑based in the veth driver.

Walking a Single Flow Through the Emulated Path

Packet Transmission and Reception

We follow a single TCP SYN from a process inside the left namespace to a listener in the right namespace.

Application send – sendmsg() copies data into an skb, sets skb->ip_summed = CHECKSUM_PARTIAL (IP and TCP checksums to be filled later), and sets skb->gso_size = 0 (no GSO yet).
veth transmit (veth1) – The veth driver’s ndo_start_xmit sees CHECKSUM_PARTIAL. Because the veth device has NETIF_F_IP_CSUM and NETIF_F_IPV6_CSUM set (default), it leaves the checksum field zero and sets skb->ip_summed = CHECKSUM_NONE. The packet is queued to the peer (veth0) via an internal skb_clone‑like handoff; no hardware DMA occurs.
Bridge ingress – The bridge receives the frame on port veth0. It performs MAC learning, looks up the destination MAC (the MAC of veth2), and forwards the frame out that port. The bridge does not modify L3/L4 fields; it merely calls dev_queue_xmit on the outgoing port.
veth transmit (veth2) – The same offload logic runs again: the veth driver sees CHECKSUM_PARTIAL, leaves the checksum zero, and transmits the skb to its peer (veth3) in the right namespace.
veth receive (veth3) – The veth receiver’s netif_receive_skb checks skb->ip_summed. Because it is CHECKSUM_NONE, the stack will later compute the checksum in ip_rcv_finish (IPv4) or ipv6_rcv (IPv6). If GRO is enabled on veth3, the receiver may attempt to merge this skb with others in the same NAPI poll cycle.
TCP stack – After checksum verification, the TCP layer processes the SYN, allocates a socket, and sends back a SYN‑ACK, which traverses the reverse path.

Role of veth Pairs in Packet Forwarding

Each veth endpoint acts as a full‑duplex pipe with zero‑copy latency under normal conditions: the transmitter simply passes a pointer to the same skb (or a clone) to the receiver’s NAPI loop. The only CPU work is the reference‑count bump and the NAPI poll scheduling. If checksum offload is active, the transmitter does not compute the checksum; the receiver does it later, adding a few hundred cycles per packet. If GRO is active, the receiver may hold the skb for up to gro_flush_timeout (default 10 ms) to attempt aggregation, adding latency but reducing per‑packet interrupt overhead.

Bridge Configuration and Packet Flow

The bridge’s forwarding decision is made in br_handle_frame_finish. It consults the forwarding database (FDB) which is initially empty, so the first packet is flooded to all ports except the ingress port. After the first exchange, the FDB learns the MACs of veth1 and veth3, so subsequent unicast frames are forwarded directly to the correct port. The bridge does not touch checksum or GRO flags; it merely forwards the skb as‑is.

Understanding Checksum and GRO Offload

Checksum Offload: Benefits and Implications

Benefit: Offloading checksum computation saves CPU cycles, especially for high‑rate small packets where the checksum is a non‑trivial fraction of processing cost.
Implication for capture: On the transmit side of a veth device (or any software device with NETIF_F_IP_CSUM), the IP header checksum field will be zero, and the TCP/UDP checksum will be the pseudo‑header sum only. Tools that verify checksums (e.g., tcpdump -vv) will mark the packet as “bad”. The packet is still valid; the kernel will compute the correct checksum just before handing the packet to the device driver for actual transmission (or, in the veth case, just before delivering it to the peer).
Kernel version note: As of Linux 5.4, veth devices expose NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | NETIF_F_GRO | NETIF_F_GSO by default. Inspect with ethtool -k veth0.

GRO Offload: Benefits and Implications

Benefit: Merges multiple small packets into a single larger skb, reducing NAPI polling cycles, interrupt handling, and socket buffer allocations. This can cut CPU usage by 30‑50 % for bulk TCP streams.
Implication for capture: If you capture on the receive side after GRO has run, you will see packets whose length exceeds the MTU (often 64 KB for TCP GRO). The inner segmentation is lost; you cannot see the original MSS‑sized segments. If you capture before GRO (e.g., by tapping the veth transmit side or disabling GRO on the receiver), you see the original segments.
Kernel version note: GRO is enabled by default on veth since Linux 4.1. The gro flush timeout can be tuned via /proc/sys/net/core/gro_flush_timeout.

Impact on Packet Capture and Analysis

When you run tcpdump -i veth0 -w capture.pcap, you are attaching to the transmit queue of the veth device. At that point:

Checksums are still partial (zero) because the veth driver has not yet asked the receiver to compute them.
GRO has not yet occurred because GRO is a receive‑side optimization; the transmit side sees the original segments as they were generated by the TCP stack.

If you instead capture on veth3 (the receive side in the right namespace) without disabling GRO, you may observe:

Correct checksums (the receiver has already computed them).
Fewer, larger packets due to GRO aggregation.

Thus, a capture that looks “broken” (bad checksums, jumbo frames) is often simply a side effect of the offload being active at the point of capture.

Troubleshooting Capture Issues

Identifying Checksum and GRO Offload Problems

Checksum symptoms – tcpdump output shows bad checksum or incorrect (-> 0) for IP/TCP/UDP fields.
GRO symptoms – You see packets with len > 1500 (or > MTU) and the TCP sequence numbers jump by more than one MSS; Wireshark may show “TCP segment of a reassembled PDU”.
Verification – Use ethtool -k <iface> to list offload features. If rx-checksumming, tx-checksumming, gro, gso, tso, ufo are on, the device is capable of offloading.

Disabling Offload Features for Capture

To obtain a “raw” capture that matches what the application actually sent/received, disable the relevant offloads on the interface you are tapping:

# Disable checksum offload (both TX and RX)
ethtool --offload veth0 rx off tx off
# Disable GRO and GSO
ethtool --offload veth0 gro off gso off tso off ufo off

After disabling, the veth driver will set skb->ip_summed = CHECKSUM_NONE on transmit and will not attempt GRO on receive, so the capture will show the original packet shapes and correct checksums (computed by the stack).

Using CLI Tools for Troubleshooting

ethtool – Query and modify offload flags, view driver statistics (ethtool -S veth0).
ip -s link – Show per‑interface packet and byte counts, drops, overruns.
tcpdump / wireshark – Capture with -vv to see checksum validation; use -T tcp to force TCP dissection.
bpftrace – Trace points like tracepoint:net:netif_receive_skb to see when GRO aggregation occurs.
Example:

# Trace GRO flush events on veth3
bpftrace -e 'tracepoint:net:netif_receive_skb /args->skb->len > 1500/ { printf("GRO packet %d bytes\\n", args->skb->len) }'

Code Examples for Configuring and Troubleshooting

Using iproute2 and ethtool for Configuration

# Create veth pair and move ends to namespaces
ip link add veth0 type veth peer name veth1
ip link add veth2 type veth peer name veth3
ip netns add left
ip netns add right
ip link set veth1 netns left
ip link set veth3 netns right
ip link set veth0 up
ip link set veth2 up
ip netns exec left ip link set veth1 up
ip netns exec right ip link set veth3 up

# Create bridge and attach veth ends
ip link add name br0 type bridge
ip link set br0 up
ip link set veth0 master br0
ip link set veth2 master br0
bridge link show

This block reproduces the setup steps; adjust interface names or namespace labels as needed for your scenario.