Packet-walk replay cases for overlay blackholes

##Introduction to Replay Scenarios Network troubleshooting in modern Linux environments often requires reproducing a failure that traverses multiple isolation boundaries: network namespaces, virtual Ethernet (veth) pairs, overlay tunnels such as VXLAN, MTU mismatches that trigger fragmentation, and policy enforcement points like iptables/nftables or eBPF programs. A replay scenario reconstructs the exact packet path of a failed flow, allowing the engineer to observe where the packet is dropped, altered, or delayed, and to validate fixes without affecting production traffic.

When a service reports timeout or connection reset, the root cause may lie anywhere along the dataplane. Guessing leads to wasted time and potentially disruptive changes. By building a deterministic replay—creating the same namespaces, veth links, VXLAN endpoints, MTU settings, and policy rules that existed at the moment of failure—you can:

• Capture packets at each hop with tcpdump or bpftrace.
• Measure latency introduced by queues, offloads, or software forwarding.
• Verify that MTU‑induced fragmentation or reassembly works as expected.
• Confirm that a policy hook (e.g., a drop, such as an iptables rule or XDP program) is not unintentionally discarding the packet.
• Validate that any fix (MTU adjustment, rule change, offload toggle) eliminates the drop without side effects.

A reproducible scenario turns an intermittent symptom into a controlled experiment, making root‑cause analysis measurable and repeatable.

Network Architecture and Components

Namespaces and Network Isolation

Linux network namespaces (netns) provide independent routing tables, ARP caches, and socket namespaces. A process bound to a namespace sees only its own interfaces and routes. Typical use cases include container runtimes (CRI‑O, containerd), CNI plugins, and test harnesses. Key commands:

# Create two namespaces
ip netns add ns-left
ip netns add ns-right

# List namespaces
ip netns show

Each namespace gets its own loopback device (lo) and can hold veth peers, bridges, or VXLAN devices. Routes and rules are isolated; a packet leaving ns-left via a veth pair enters the peer in the default namespace (or another ns) and is subject to that namespace’s forwarding decisions.

Veth Pairs and Virtual Networking

A veth pair is a virtual Ethernet cable: two interfaces (vethA and vethB) where traffic transmitted on one appears immediately on the other, preserving Ethernet headers. They are the primary mechanism to connect namespaces to each other or to a bridge/OVS switch.

Creation:

ip link add veth-left type veth peer name veth-right
ip link set veth-left netns ns-left
ip link set veth-right netns ns-right
ip netns exec ns-left ip link set dev veth-left up
ip netns exec ns-right ip link set dev veth-right up

Key properties:

• MTU defaults to 1500 but can be changed independently on each end (ip link set dev veth-left mtu 1400).
• No checksum offload by default; the kernel treats them as regular netdevices.
• Queuing discipline (qdisc) can be attached (e.g., pfifo_fast, fq_codel) to shape or measure latency.

VXLAN Encapsulation and Tunneling

VXLAN (Virtual Extensible LAN) encapsulates Ethernet frames in UDP (port 4789 by default) with an 8‑byte VXLAN header containing a 24‑bit VNI (VLAN Identifier). It enables L2 overlay over an L3 underlay, useful for connecting containers across hosts or separating tenant traffic.

Typical dataplane flow for an outbound packet from a container:

1. Ethernet frame (dst MAC of remote VTEP) → VXLAN encap (add UDP + VXLAN header) → outer IP/UDP header (src = local VTEP IP, dst = remote VTEP IP) → transmitted via physical NIC.
2. On the remote host, the outer headers are stripped, the inner Ethernet frame is delivered to the appropriate bridge or veth.

Linux VXLAN device creation (using ip link):

# On host A (VTEP 10.0.0.1, VNI 42)
ip link add vxlan42 type vxlan id 42 dev eth0 dstport 4789 \
    local 10.0.0.1 remote 10.0.0.2
ip link set dev vxlan42 up
ip addr add 10.244.42.1/24 dev vxlan42   # optional L3 address for health checks

Important notes:

• The outer UDP payload size = inner Ethernet frame + VXLAN header (8 bytes). If the inner frame is 1500 bytes, the outer UDP packet is 1508 bytes (+20 IP +8 UDP = 1536 bytes total).
• NICs may offload VXLAN encapsulation/decapsulation if they support udp_tunnel or lx features (ethtool -k eth0 shows tx-vxlan-offload, rx-vxlan-offload).
• If the underlying MTU is smaller than the required outer packet size, fragmentation occurs (see MTU Edges section).

MTU Edges and Packet Fragmentation

Maximum Transmission Unit (MTU) defines the largest IP packet size that can be sent without fragmentation on a link. Common values: Ethernet 1500, VXLAN overlay often requires ≤1450 to accommodate the 50‑byte VXLAN/UDP/IP overhead when the underlay MTU is 1500.

When a packet exceeds the egress interface MTU, the kernel (or NIC if offload enabled) performs IP fragmentation:

• The original packet is split into fragments, each with its own IP header (flags: MF except last, offset field).
• Reassembly occurs at the final destination host (not intermediate routers) unless the path includes devices that perform transparent fragmentation/reassembly (rare in modern data centers).

Fragmentation introduces overhead:

• Additional IP header per fragment (20 bytes for IPv4, 40 for IPv6).
• Potential out‑of‑order delivery, increased reassembly latency, and higher CPU cost.
• Some middleboxes or security policies drop fragments (e.g., iptables -p icmp --icmp-type fragmentation-needed -j DROP).

To discover MTU mismatches, use tracepath or ping -M do -s <size>:

# From ns-left, test path to ns-right via veth pair
ip netns exec ns-left ping -M do -s 1472 10.200.200.2   # 1472 + 28 ICMP header = 1500

If the packet is too large, you’ll see “Frag needed and DF set” ICMP messages.

Policy Hooks and Network Policies

Policy hooks are points in the packet path where the kernel can inspect, modify, or discard packets based on rules. Main mechanisms:

A packet may be dropped at any of these points. For example, an iptables rule in the FORWARD chain with -j DROP will silently discard a transiting packet unless logging is enabled (-j LOG). Similarly, an XDP program returning XDP_DROP aborts processing before the packet reaches the stack.

Understanding which hook is active requires inspecting the rule set (iptables -S, nft list ruleset, tc filter show dev <iface> egress, ip -d link show dev <iface> for XDP programs).

Building Replay Scenarios

Identifying Failed Flows and Packet Loss

Start by gathering symptoms: timeout, connection reset, retransmits visible in application logs or ss -s. Use tcpdump on both ends to see if packets leave the source but never arrive at the destination.

Example: a client in ns-left tries to HTTP GET a server in ns-right via VXLAN. Capture:

# In ns-left, capture on veth-left
ip netns exec ns-left tcpdump -i veth-left -w left.pcap

# In ns-right, capture on veth-right
ip netns exec ns-right tcpdump -i veth-right -w right.pcap

If left.pcap shows SYN sent but right.pcap shows nothing, the packet is dropped somewhere between the veth pair and the VXLAN decap.

Creating Replay Scenarios with Veth Pairs

Replicate the exact namespace topology observed during the failure. If the original used a bridge, recreate it:

# Create bridge in default ns
ip link add br0 type bridge
ip link set dev br0 up

# Attach veth peers to bridge
ip link set veth-left master br0
ip link set veth-right master br0
ip link set dev veth-left up
ip netns exec ns-left ip link set dev veth-left up
ip netns exec ns-right ip link set dev veth-right up

Assign IP addresses appropriate to the scenario (e.g., 10.0.0.0/24 for underlay, 10.1.0.0/24 for overlay).

Configuring VXLAN Encapsulation and Decapsulation

Set up VXLAN endpoints matching the original VNI and remote IPs. Ensure the underlay MTU is set correctly to avoid unwanted fragmentation unless you intend to test it.

# Underlay interfaces (assume eth0 is the physical or dummy interface)
ip link set dev eth0 mtu 1500   # adjust if needed

# VXLAN device in ns-left (acting as VTEP A)
ip netns exec ns-left ip link add vxlan10 type vxlan id 10 dev eth0 dstport 4789 \
    local 10.0.0.1 remote 10.0.0.2
ip netns exec ns-left ip link set dev vxlan10 up
ip netns exec ns-left ip addr add 10.1.0.1/24 dev vxlan10

# VXLAN device in ns-right (VTEP B)
ip netns exec ns-right ip link add vxlan10 type vxlan id 10 dev eth0 dstport 4789 \
    local 10.0.0.2 remote 10.0.0.1
ip netns exec ns-right ip link set dev vxlan10 up
ip netns exec ns-right ip addr add 10.1.0.2/24 dev vxlan10

Connect the veth ends to the VXLAN device (or to a bridge that includes the VXLAN):

ip netns exec ns-left ip link set dev veth-left master vxlan10
ip netns exec ns-right ip link set dev veth-right master vxlan10

Simulating MTU Edges and Packet Fragmentation

To test MTU‑induced fragmentation, deliberately lower the MTU on an underlay link or on the VXLAN device.

# Reduce underlay MTU to 1400 bytes (forces fragmentation if outer packet >1400)
ip link set dev eth0 mtu 1400
ip netns exec ns-left ip link set dev eth0 mtu 1400
ip netns exec ns-right ip link set dev eth0 mtu 1400

Now a 1500‑byte inner Ethernet frame (plus 28‑byte VXLAN/UDP/IP overhead) will exceed 1400, causing IP fragmentation. Verify with:

ip netns exec ns-left ping -M do -s 1450 10.1.0.2   # 1450 + 28 = 1478 > 1400 -> frag needed

You should see “Frag needed and DF set” ICMP messages unless you clear the DF flag (-M want).

Implementing Policy Hooks and Network Policies

Insert a policy that drops packets with a specific characteristic (e.g., TCP dport 80) to emulate a firewall rule that caused the failure.

Using nftables (preferred over legacy iptables):

# Create a table and chain in the default namespace (or specific ns)
nft add table inet filter
nft add chain inet filter forward { type filter hook forward priority 0 \; }

# Drop TCP packets to port 80
nft add rule inet filter forward tcp dport 80 drop

If you want to log before dropping:

nft add rule inet filter forward tcp dport 80 log prefix "FW-DROP: " drop

Alternatively, attach an XDP program that drops packets with a certain VNI:

// simple_drop_xdp.c
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

SEC("xdp")
int xdp_drop_vni(struct xdp_md *ctx) {
    void *data_end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;
    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)
        return XDP_ABORTED;
    /* Example: drop if VNI in VXLAN header equals 42 (simplified) */
    return XDP_DROP;
}

Compile and load:

clang -O2 -target bpf -c simple_drop_xdp.c -o simple_drop_xdp.o
ip link set dev veth-left xdp obj simple_drop_xdp.o sec xdp

With the replay scenario built, you can now iterate: capture, inspect, adjust MTU or rules, and verify that the packet successfully traverses the path. This methodical approach isolates the exact point of failure and validates the fix in a safe, reproducible environment.