How much reordering can Linux TCP actually absorb

Introduction to Benchmarking ECMP Jitter

Understanding ECMP and SACK

Equal‑Cost Multi‑Path (ECMP) routing spreads flows across multiple paths that have identical routing cost. Per‑packet load‑balancing (often based on a 5‑tuple hash) can cause packets of the same TCP flow to traverse paths with different propagation delays. The resulting packet reordering appears at the receiver as a sequence gap that TCP interprets as loss unless the loss‑recovery mechanism can distinguish reordering from genuine loss.

Selective Acknowledgment (SACK) lets the receiver inform the sender about non‑contiguous blocks of received data, allowing the sender to retransmit only the missing segments. Modern loss‑recovery extensions such as RACK (Recent ACK) and FACK (Forward ACK) add a time‑based reordering window to avoid mistaking reordering for loss. However, these mechanisms are bounded by the sysctl net.ipv4.tcp_reordering (default = 3). When the reordering depth exceeds this threshold, TCP may trigger a spurious retransmission even though SACK reports the data as received.

Importance of Benchmarking Packet Reordering

Network operators need to know at what point ECMP‑induced jitter overwhelms the loss‑recovery heuristics and starts generating avoidable retransmissions. Those retransmissions waste bandwidth, increase CPU cycles for retransmit processing, and can inflate latency tail‑metrics. A repeatable benchmark that varies reordering depth (how many packets ahead a later packet can arrive) and burst shape (the temporal distribution of those reordered packets) provides a controlled way to measure the exact break‑even point where SACK/RACK stop masking ECMP jitter and start causing harm.

Designing the Benchmark

Defining Packet Reordering Depth and Burst Shape

• Reordering depth (D) – the maximum number of packets by which a later packet can precede an earlier one in the receiver’s view. In Linux tc netem this is expressed as the gap parameter (e.g., gap 3 means a packet can be reordered up to three positions ahead).
• Burst shape (B) – the statistical distribution of when reordered packets occur within a traffic stream. We characterize it by two dimensions:
  1. Probability of reorder (p) – the chance any given packet is subject to reordering.
  2. Temporal correlation (c) – the likelihood that a reordered packet is followed by another reordered packet (modeled by netem’s correlation option).

A bursty shape (high c) creates clusters of reordered packets, which is more stressful for TCP’s reordering detection than a uniform Poisson distribution.

Selecting Relevant Network Parameters

Choosing a Suitable Benchmarking Tool

• Traffic generator – iperf3 in TCP mode provides built‑in retransmission statistics via the -J JSON output and is easy to script. For >10 Gbps rates, replace with pktgen (kernel) or MoonGen/DPDK, but the core methodology stays identical.
• Reordering injector – Linux tc netem with the reorder option can emulate both depth and burst shape on a single interface.
• Measurement collection – ss -ti (TCP info) gives per‑socket retransmits, tcp_info fields (tcpi_retransmits, tcpi_sacked, tcpi_lost). System‑wide counters from /proc/net/snmp (TcpRetransSegs, TcpOutRets) validate socket‑level data.
• CPU/utilization – pidstat -p <iperf3-pid> 1 or perf stat to capture cycles spent in retransmit handling.

The benchmark will be structured as a repeatable experiment:

1. Baseline – no reordering, SACK + RACK on, default tcp_reordering.
2. Sweep – vary D (1‑20) and B (p = 0.1‑0.5, c = 0‑0.9) while holding load constant.
3. Observe – retransmit count, goodput, CPU cycles, and tail latency.
4. Optimized variant – increase tcp_reordering or disable SACK/RACK to see if retransmits drop.
5. Trade‑off – measure any loss in recovery speed when genuine loss is introduced.

The experiment ends when we can point to a specific (D,B) pair where retransmits rise above the baseline and the increase correlates with CPU cycles spent in tcp_retransmit_skb.

Implementing the Benchmark

Setting Up the Network Environment

# Two hosts: sender (s) and receiver (r) connected via a switch with ECMP enabled
# Ensure the path is symmetric and uses per‑packet load‑balancing
# Disable unnecessary offloads to see raw packet reordering
ethtool -K eth0 gro off lro off tso off gso off rx off tx off

# Set base RTT (add 20 ms of static delay on both ends for a 40 ms RTT)
tc qdisc add dev eth0 root netem delay 20ms

On the receiver, open a listening iperf3 server:

iperf3 -s -J --logfile r_iperf.json

On the sender, start the client with a fixed bandwidth (e.g., 300 Mbps) to avoid link saturation:

iperf3 -c <receiver_ip> -t 60 -b 300M -J --logfile s_iperf.json

Configuring SACK and Loss Recovery Mechanisms

# Verify defaults
sysctl net.ipv4.tcp_sack
sysctl net.ipv4.tcp_rack
sysctl net.ipv4.tcp_reordering

# Example: turn SACK off, keep RACK on
sysctl -w net.ipv4.tcp_sack=0
sysctl -w net.ipv4.tcp_rack=1
# Example: increase reordering threshold
sysctl -w net.ipv4.tcp_reordering=10

All changes are applied before starting the traffic generator and persist for the duration of the test.

Generating Packet Reordering with Varying Depths and Burst Shapes

Using tc netem on the outbound interface of the sender (or inbound on the receiver; symmetry matters less as long as reordering is introduced somewhere on the path):

# Helper function to apply a given reorder profile
apply_reorder() {
    local depth=$1   # D
    local prob=$2    # p (0‑1)
    local corr=$3    # c (0‑1)
    tc qdisc change dev eth0 root netem delay 20ms \
        reorder $prob% $corr% gap $depth
}

# Sweep example (bash loop)
for D in 1 2 3 5 10 15 20; do
    for p in 0.1 0.3 0.5; do
        for c in 0.0 0.5 0.9; do
            apply_reorder $D $p $c
            sleep 5   # let the queue settle
            iperf3 -c <receiver_ip> -t 20 -b 300M -J --logfile \
                results_D${D}_p${p}_c${c}.json
        done
    done
done

The gap $depth argument tells netem how many packets ahead a reordered packet may be placed. The reorder $prob% $corr% arguments control the probability and correlation of the reorder event, thus shaping the burst.

Running the Benchmark

Executing the Benchmark with Sample Code/CLI Examples

A compact Python driver that orchestrates the sweep, collects TCP info, and aggregates results:

#!/usr/bin/env python3
import subprocess, time, json, os, sys

IFACE = "eth0"
SERVER = "10.0.0.2"
BASE_DELAY = "20ms"
RESULTS_DIR = "results"
os.makedirs(RESULTS_DIR, exist_ok=True)

def set_reorder(depth, prob, corr):
    cmd = ["tc", "qdisc", "change", "dev", IFACE, "root", "netem",
           "delay", BASE_DELAY,
           "reorder", f"{prob}%", f"{corr}%", "gap", str(depth)]
    subprocess.run(cmd, check=True)

def run_iperf(label):
    out_file = os.path.join(RESULTS_DIR, f"{label}.json")
    cmd = ["iperf3", "-c", SERVER, "-t", "20", "-b", "300M",
           "-J", "--logfile", out_file]
    subprocess.run(cmd, check=True)
    with open(out_file) as f:
        return json.load(f)

def get_tcp_stats(pid):
    # Use ss to retrieve per‑socket info
    out = subprocess.check_output(
        ["ss", "-ti", "state", "established", "( dport = :5201 )"],
        text=True
    )
    # Parse output as needed …
    return out

if __name__ == "__main__":
    # Example sweep
    for D in [1, 2, 3, 5, 10, 15, 20]:
        for p in [0.1, 0.3, 0.5]:
            for c in [0.0, 0.5, 0.9]:
                set_reorder(D, p, c)
                time.sleep(5)  # settle
                data = run_iperf(f"D{D}_p{int(p*100)}_c{int(c*100)}")
                stats = get_tcp_stats(<iperf3-pid>)
                # Store or print results …
                print(f"D={D}, p={p}, c={c} -> {data}")

The driver applies each reorder profile, waits for the queue to settle, runs an iperf3 test, collects JSON output, and retrieves per‑socket TCP statistics via ss -ti. Adjust the parsing of get_tcp_stats to extract fields such as tcpi_retransmits for further analysis.

End of benchmark description.