How much reordering becomes visible pain

Introduction

Packet reordering occurs when packets transmitted in a specific sequence arrive at the receiver out of that order. In a TCP stack, the receiver expects a strictly increasing sequence number; any gap triggers duplicate acknowledgments (dup‑acks) and may cause the sender to invoke fast retransmit or, if the gap persists, a spurious retransmission timeout (RTO). The TCP specification tolerates a limited amount of reordering (controlled by net.ipv4.tcp_reordering) before treating it as loss. Understanding where that tolerance ends and real loss begins is essential for performance‑critical workloads that cannot afford unnecessary retransmissions or inflated latency.

Benchmarking provides a repeatable, measurable baseline against which any change—kernel tunables, NIC offloads, or application‑level adjustments—can be evaluated. Without a controlled experiment that injects known reordering patterns, observations of dup‑acks or spurious retransmits remain anecdotal. A benchmark isolates the variable (reordering) while holding throughput, latency, and CPU load constant, allowing you to pinpoint the exact point at which the TCP stack’s loss detection logic misfires. This is the foundation for any performance‑tuning effort that claims to reduce unnecessary retransmissions.

TCP Stack Behavior

Protocol Basics and Reordering Handling

TCP delivers a byte stream by assigning each byte a monotonically increasing sequence number. The receiver sends an ACK for the next expected sequence number. If a segment arrives with a higher sequence number than expected, the receiver buffers it (if out‑of‑order buffering is enabled) and sends a dup‑ack that repeats the last in‑order ACK. The sender counts dup‑acks; upon reaching the duplicate threshold (default 3), it assumes a loss and performs fast retransmit. If the missing segment never arrives before the RTO expires, a spurious retransmission occurs.

Key kernel parameters:

net.ipv4.tcp_reordering – maximum reordering distance tolerated before treating a gap as loss (default 3).
net.ipv4.tcp_early_retrans – enables early retransmit after fewer dup‑acks when reordering is suspected low.
net.ipv4.tcp_fack – enables forward acknowledgment to better measure outstanding data during partial acknowledgments.

Dup‑acks vs. Spurious Retransmissions

A dup‑ack is an ACK whose acknowledgment number equals the highest contiguous sequence number received so far. When the receiver sees a gap, it repeatedly sends the same ACK for each out‑of‑order segment that arrives. The sender interprets a streak of dup‑acks as evidence that a segment was lost, not merely delayed.

A spurious retransmission happens when the sender retransmits a segment that was never lost; the original segment arrives later, rendering the retransmission unnecessary. This wastes bandwidth, increases CPU overhead (extra checksum, segmentation, and transmission), and can trigger congestion control reactions (e.g., cwnd reduction) even though no real loss occurred.

Real Path Loss vs. Packet Reordering

Real path loss results from packet drops in the network (congestion, hardware errors, etc.). The TCP stack treats loss as a signal to reduce sending rate. Reordering, by contrast, does not imply loss; the missing segment is merely delayed. The distinction matters because reacting to reordering as loss degrades throughput unnecessarily. The TCP stack’s heuristic (tcp_reordering) attempts to differentiate the two, but it is a static threshold that may be inappropriate for high‑bandwidth, high‑delay paths where larger reordering bursts are common.

Benchmarking Controlled Packet Reordering

Methodologies for Introducing Reordering

The most deterministic way to inject reordering on a Linux host is via the tc (traffic control) netem qdisc with the delay and correlation options, or via iptables/nftables with the statistic module to probabilistically reorder packets. For finer granularity, eBPF/XDP programs can reorder packets at NIC ingress by storing them in a per‑CPU ring and emitting them after a controlled delay.

Common approaches:

tc netem delay distribution – adds a random delay to a subset of packets, causing them to overtake earlier packets.
tc netem reorder – directly reorders a percentage of packets by a specified time delay.
eBPF/XDP map‑based reorder – stores packets in a hash keyed by sequence number and releases them after a timeout.
Scapy‑based traffic generation – builds packets with intentional sequence number jumps.

Tools and Techniques for Measuring TCP Behavior

Traffic generator: iperf3 (TCP) or netperf for controlled throughput and latency measurement.
Packet capture: tcpdump with -w to pcap files for offline analysis; tshark for real‑time filtering.
Kernel statistics: ss -i (shows TCP info including retransmits, dup‑acks, pacing rate), /proc/net/snmp (TCP MIB), /proc/net/netstat.
eBPF tracing: tracepoint:tcp:tcp_retransmit_skb, tracepoint:tcp:tcp_rcv_space_adjust, or custom kprobes on tcp_ack.
CPU overhead: perf stat -e cycles,instructions,cache-references,cache-misses while running the benchmark.

Example Benchmark Script

# ---- on both hosts -------------------------------------------------
# Ensure tc and iperf3 are installed
sudo apt-get update && sudo apt-get install -y iproute2 iperf3

# ---- on server (s) -------------------------------------------------
# Start iperf3 server in background
iperf3 -s -D   # daemonize
SERVER_PID=$!

# ---- on client (c) -------------------------------------------------
# 1. Baseline: no reordering
echo "Baseline run (no reorder)"
iperf3 -c $SERVER_IP -t 30 -P 4 -i 5 > baseline.txt

# 2. Introduce reordering on the client‑side egress (outgoing) interface
#    Reorder 10% of packets with a 200 µs delay (enough to cause overtaking)
sudo tc qdisc add dev eth0 root netem reorder 10% 200us

# 3. Run test with reordering
echo "Run with reordering"
iperf3 -c $SERVER_IP -t 30 -P 4 -i 5 > reorder_10pct.txt

# 4. Collect TCP stats during the test (run in another terminal)
#    Show retransmits, dup‑acks, and RTO every 5 s
watch -n 5 "ss -i state established '( dport = :5201 )' | awk '/retrans/ || /dup/'"

# 5. Cleanup
sudo tc qdisc del dev eth0 root
kill $SERVER_PID

What to look for in the output:

baseline.txt: retransmits ≈ 0, dup‑acks ≈ 0.
reorder_10pct.txt: increase in the retrans column (fast retransmits) and/or timeout column (spurious RTO).
ss -i output: fields like retrans:2/0, dupcks:5, rto:204 indicate the TCP stack’s reaction.

To vary the reordering distance, adjust the netem parameters (reorder 20% 500us) and observe the point at which retransmits climb sharply—this approximates the effective tcp_reordering threshold for your NIC and kernel version.

Identifying Dup‑acks and Spurious Retransmissions

Analyzing TCP Dump Outputs

In a pcap, a dup‑ack appears as:

13:45:12.123456 IP c.54321 > s.5201: Flags [.], ack 1001, win 229, length 0
13:45:12.124001 IP c.54321 > s.5201: Flags [.], ack 1001, win 229, length 0   # dup
13:45:12.124500 IP c.54321 > s.5201: Flags [.], ack 1001, win 229, length 0   # dup

The acknowledgment number (ack 1001) stays constant while new data segments (with higher sequence numbers) arrive. Use tcpdump -nn -r capture.pcap 'tcp[tcpflags] & tcp-ack != 0 and tcp[13] & 0x10 != 0' to isolate ACKs, then count consecutive identical ack numbers.

A spurious retransmission appears as a retransmitted segment whose sequence number falls within the current receive window but whose data was already ACKed:

13:45:12.130000 IP s.5201 > c.54321: Flags [P.], seq 1001:1101, ack 2001, win 229, length 100
...
13:45:12.135000 IP s.5201 > c.54321: Flags [P.], seq 1001:1101, ack 2001, win 229, length 100   # retransmit

If the second packet’s seq matches an earlier packet that was already acknowledged, it is spurious.

Using Wireshark for Visualization

Load the pcap → Statistics → TCP Stream Graph → Round Trip Time Graph – look for spikes in RTT that coincide with out‑of‑order arrivals.
Statistics → TCP Stream Graph → Throughput Graph – a drop in throughput often follows a burst of dup‑acks.
Analyze → Expert Information – Wireshark flags “Previous segment lost” (potential spurious retransmit) and “Duplicate ACK #n”.
TCP Analysis → Retransmission – right‑click a packet → “Mark as Retransmission” to see the original and retransmitted pair side‑by‑side.
Use the display filter tcp.analysis.retransmission to show only retransmits; combine with tcp.analysis.duplicate_ack_num > 0 to see dup‑acks.

CLI Examples for Monitoring TCP Stack Behavior

Per‑connection stats via ss:

ss -i state established '( dport = :5201 )' | awk '
  /retrans/ {retrans=$2}
  /dupcks/  {dup=$2}
  END {print "retrans:",retrans,"dup‑acks:",dup}
'

eBPF tracepoint for fast retransmit:

sudo tracepoint -e tcp:tcp_retransmit_skb -a '
@[comm, pid] = count();
'

Measure CPU cycles spent in TCP retransmit handling:

sudo perf record -e cycles:k -g -- sleep 30   # run during iperf3 test
sudo perf report | grep tcp_retransmit_skb

Troubleshooting Packet Reordering Issues

Common Causes of Packet Reordering in Networks

Load‑balancing across multiple paths (ECMP, LAG) with per‑packet hashing that varies due to header changes (e.g., TCP timestamps, UDP ports).
Asymmetric routing where forward and reverse paths differ, causing ACKs to travel a different route than data.
NIC features such as receive-side scaling (RSS) or multi‑queue where packets are dispatched to different CPUs and reordered by software scheduling.
QoS/policing that introduces variable delay (e.g., token bucket shaping) on a subset of flows.
Hardware offloads like GRO/LRO that may reorder segments during coalescing.
VPN/tunnel encapsulation where inner packets experience different latency than outer packets.