veth is not free just because it is virtual

Introduction to veth Performance

A virtual Ethernet (veth) pair acts like a software pipe: transmitting on one end delivers the packet to the receive queue of its peer. Each end is a regular struct net_device with its own queues, NAPI structures, and GRO cells. The veth driver implements the standard net_device ops but does not offload work to hardware; all processing happens in the kernel’s software datapath.

Common Misconceptions

• “veth is zero‑cost plumbing.”
  Although no physical NIC is involved, each packet triggers an skb_clone, skb_orphan, and two passes through the receive path (one per side), causing allocation, reference‑count manipulation, and GRO work.
• “Increasing txqueuelen eliminates drops.”
  The tx queue length only controls how many packets can be queued before the kernel returns NETDEV_TX_BUSY; it does not reduce per‑packet CPU overhead or GRO boundary costs.
• “GRO on veth is free.”
  GRO still consumes CPU cycles to compute hashes, check compatibility, and merge skbs. A flush forced by a GRO boundary can be more expensive than processing packets individually.

Understanding veth Overhead

SKB Allocation and Deallocation

When an application sends data through a socket bound to a veth interface, the kernel creates an skb via alloc_skb() (or __alloc_skb() for DMA‑aware allocations). The allocation draws from the skbuff_head_cache slab, backed by the general page allocator.

Allocation steps

1. Allocate a struct sk_buff (~256 B on x86_64) from the slab.
2. Allocate a data buffer of size len + sizeof(struct skb_shared_info) (via kmalloc or page allocation if > PAGE_SIZE).
3. Initialize skb->head, skb->data, skb->tail, skb->end, and set the reference count to 1.
4. Set skb->dev to the sending net_device (the veth end).

In the veth transmit path (veth_xmit):

skb = skb_clone(skb, GFP_ATOMIC);
if (!skb)
    return NETDEV_TX_BUSY;
skb_orphan(skb);
skb->dev = veth->peer;
netif_rx(skb);
return NETDEV_TX_OK;

Each outbound packet therefore incurs one skb_clone (allocates a new skb sharing the data buffer) and an skb_orphan (makes the clone the sole owner). The original skb is freed when the caller’s reference count drops, so each packet results in two allocations and two frees (one pair per veth end).

Deallocation
After the peer’s receive path processes the cloned skb, it calls consume_skb() or kfree_skb():

• consume_skb() decrements the reference count; if it reaches zero, it invokes kfree_skb().
• kfree_skb() returns the data buffer to the appropriate slab/page cache and frees the skb structure.

Queueing and Buffering

veth does not implement its own queueing discipline; it relies on the peer’s qdisc for transmission scheduling. By default each veth end inherits the root qdisc of its network namespace (usually noop or pfifo_fast if a default qdisc is set). The txqueuelen controls the size of the netdev’s transmission queue (dev->tx_queue_len). When the queue fills, netif_tx_stop_queue() is called and subsequent ndo_start_xmit returns NETDEV_TX_BUSY.

Key points:

• The veth driver merely passes the cloned skb to netif_rx() of the peer; it performs no packet scheduling.
• Backpressure propagates via the tx queue: if the peer’s receive path cannot keep up, its queue fills, causing the transmitter to stall.
• No hardware offload (TC, XDP, etc.) is involved unless explicitly attached to the veth netdev.

Buffering strategies

• Receive side: The peer’s NAPI poll routine (napi_poll) pulls skbs from its receive queue (dev->rx_queue). If NAPI is disabled (e.g., ethtool -C <dev> rx-usecs 0), the kernel falls back to interrupt‑driven receive, increasing per‑packet overhead.
• Transmit side: The clone is placed directly onto the peer’s receive queue; there is no separate transmit buffer beyond the tx queue length.
• Memory pressure: High transmit rates can exhaust the skbuff_head_cache slab, leading to allocation failures (__alloc_skb returns NULL) and NETDEV_TX_BUSY drops.

GRO Boundaries and Their Impact

What are GRO Boundaries

Generic Receive Offload (GRO) merges multiple incoming skbs into a larger skb to reduce per‑packet processing overhead. A GRO boundary occurs when the GRO engine cannot merge an incoming skb with the current aggregation because of incompatibility (different flow hash, VLAN tag, TCP flags, checksum offload status, or exceeding gro_max_size). When a boundary is hit, the engine flushes the aggregated skb up the stack and starts a new aggregation.

In veth, each end maintains its own struct gro_cells (one per NAPI instance). The GRO decision is made after the veth receive path (veth_gro_receive) but before the skb is delivered to higher layers (IP, TCP, etc.).

How GRO Boundaries Affect Performance

When a boundary is crossed:

1. The current aggregated skb is passed to netif_receive_skb() (or netif_rx() if GRO is disabled), incurring header adjustments, checksum verification, and protocol processing.
2. A new aggregation starts with the incoming skb.
3. The flush incurs the full cost of processing a regular packet (IP/TCP/UDP header parsing, socket lookup, etc.) for each skb in the flushed aggregate.

If traffic patterns cause frequent boundaries (mixed TCP/UDP, VLAN tag changes, asymmetric checksum offload), the GRO benefit diminishes and CPU cost approaches that of processing each packet individually.

GRO Boundary Optimization

• Enable GRO on both ends: ethtool -K veth0 gro on (and the peer). If one side has GRO disabled, every packet triggers a flush.
• Uniform flow characteristics: Keep VLAN IDs, TCP/UDP ports, and checksum offload consistent across the stream to maximize merge likelihood.
• Increase gro_flush_timeout (if available): Some drivers expose a sysfs parameter to delay flushing, allowing more time for compatible packets to arrive. For veth, the timeout is controlled via the NAPI poll interval; reducing poll frequency (/sys/class/net/<dev>/gro_flush_timeout) can decrease flush frequency at the cost of latency.
• Adjust gro_max_size: Raising the maximum aggregated size (ethtool -G <dev> gro-max-size <bytes>) permits larger merges, reducing the number of flushes for bulk traffic.

Per‑Packet CPU Work and Its Effects

Receive Path CPU Work

When a cloned skb arrives at the peer’s veth receive path:

1. netif_rx() (or netif_receive_skb() if NAPI is active) places the skb onto the per‑CPU backlog.
2. NAPI poll (napi_poll) invokes veth_gro_receive():
   • Computes a flow hash (skb_get_hash()).
   • Checks GRO compatibility (same hash, same IP proto, same TCP flags, etc.).
   • If compatible, calls gro_receive() to merge the skb into the current aggregation.
   • If incompatible, triggers a GRO boundary flush as described above.

Each step involves reference‑count manipulation, hash calculations, and conditional branching, all of which consume CPU cycles. The cumulative cost per packet—especially when GRO boundaries are frequent—sets the practical ceiling for veth‑based throughput, far above the negligible cost implied by the “zero‑cost plumbing” myth.