Measure random reorder, gap, and duplicate settings under fixed bandwidth to show when netem stops emulating a bad link and starts inventing recovery behavior your production path will never create.

Introduction to Network Emulation with Netem

Netem (Network Emulator) is a Linux traffic‑control (tc) qdisc that injects impairments—delay, loss, duplication, reordering, and corruption—into packets. It works at the dataplane level, manipulating sk_buffs before they reach the driver (egress) or after they are received (ingress). Common uses include:

• Reproducing WAN characteristics in a lab for application performance testing.
• Validating timeout, retransmission, and congestion‑control logic in transport protocols.
• Stress‑testing overlay networks, VPNs, and SD‑WAN solutions under controlled packet‑level perturbations.
• CI pipelines that require deterministic network fault injection.

Because netem integrates with the kernel’s QoS framework, it can be stacked with other qdiscs (e.g., HTB for bandwidth shaping) and inspected with standard tc tools, making it a low‑overhead primitive for repeatable experiments.

Why Emulate Real‑World Conditions?

Production paths rarely exhibit ideal FIFO behavior. Microbursts, out‑of‑order delivery from asymmetric routing, link‑layer retransmissions, and occasional duplicate frames are observable on real links. Applying only static delay or loss misses timing sensitivities that trigger spurious retransmissions, premature fast‑retransmit, or unnecessary congestion‑window reductions. Accurate emulation of random reorder, gap (jitter), and duplication lets you observe whether your stack’s recovery mechanisms are robust or merely lucky under benign conditions.

Understanding Netem Parameters

Random Reorder

Netem’s reorder model probabilistically holds a packet for a configurable correlation time and then releases it after a distance (in packets) delay.

Full specification: reorder <percentage>% <gap> correlation <percentage>%.
When gap is N, a packet may be delayed up to N other packets before transmission, creating a reordering window of N packets. Netem uses a simple ring buffer; if the buffer fills, excess packets are dropped, which can unintentionally add loss.

Gap (Reorder Distance)

In netem terminology, gap is not a standalone option; it is the sub‑parameter of the reorder model that defines the maximum reorder distance. It directly influences the maximum observable packet displacement and therefore the maximum reorder‑induced RTT variance.

Duplicate

The duplicate model emits a copy of a packet with a given probability.

Duplication does not alter packet contents; it increases offered load on the egress side. At high rates, extra packets can saturate the shaping qdisc or NIC transmit ring, leading to unexpected drops.

Fixed Bandwidth Configuration

Netem does not shape bandwidth; it merely impairs packets. To fix the available bandwidth, attach a shaping qdisc (e.g., HTB or CBQ) above netem in the tc hierarchy:

root qdisc: HTB (rate 100mbit)
          |
          +-- netem (delay, loss, reorder, duplicate)

The HTB class caps the bitrate; netem then sees a deterministic arrival process, allowing you to isolate the effect of impairment parameters without bandwidth variability confounding results.

Configuring Netem for Emulation

Installing and Setting Up Netem

Netem is part of the iproute2 package, present on virtually all modern distributions.

# Verify installation
tc -v show
# Expected output: iproute2-<version>

# Install if missing (Debian/Ubuntu)
sudo apt-get update && sudo apt-get install -y iproute2

# Install if missing (RHEL/CentOS/Fedora)
sudo dnf install -y iproute2   # or yum on older releases

No kernel modules are required; netem is built‑in (CONFIG_NET_SCH_NETEM).

Basic Netem Commands

Attach netem to an interface (e.g., eth0) under a parent HTB class:

# 1. Create root HTB with 100 Mbit ceiling
sudo tc qdisc add dev eth0 root handle 1: htb default 11

# 2. Create a class that limits to 50 Mbit (fixed bandwidth for the test)
sudo tc class add dev eth0 parent 1: classid 1:10 htb rate 50mbit ceil 50mbit

# 3. Attach netem as a child of that class
sudo tc qdisc add dev eth0 parent 1:10 handle 10: netem ...

Example: Configuring Random Reorder

# 10% reorder, gap up to 5 packets, 25% correlation
sudo tc qdisc change dev eth0 parent 1:10 handle 10: netem \
    reorder 10% 5 correlation 25%

Example: Configuring Duplicate and Reorder

# 5% duplicate, 15% reorder with gap 3 packets, no correlation
sudo tc qdisc change dev eth0 parent 1:10 handle 10: netem \
    duplicate 5% reorder 15% gap 3

Verify the active qdisc:

tc -s qdisc show dev eth0

Output lists the HTB root, the class, and the netem leaf with statistics (packets processed, dropped, duplicated, reordered).

Troubleshooting Netem Emulation

Common Issues

Debugging with CLI Tools

• tc statistics – tc -s qdisc show dev eth0 gives packets, bytes, dropped, duplicated, reordered counts.

• netem debug – Enable kernel tracing via tracefs:

  echo 1 > /sys/kernel/debug/tracing/events/net/netem/enable
  cat /sys/kernel/debug/tracing/trace_pipe | grep netem

  This shows each packet’s decision (delay, loss, duplicate, reorder) with timestamps.

• Packet generators – Use pktgen to produce a known packet stream and compare input vs output.

Analyzing Traffic with Tcpdump

Capture on both sides of the netem point (if using a veth pair) or on the same interface with -i any and filter by the netem mark (if set). Example to quantify reorder:

# Capture on the egress side
sudo tcpdump -i eth0 -w reorder.pcap

# Post‑process with a small Python script (requires pyshark)
python3 - <<'PY'
import pyshark
cap = pyshark.FileCapture('reorder.pcap', keep_packets=False)
seq = {}
reorder_events = 0
for pkt in cap:
    if 'TCP' in pkt:
        try:
            seq_num = int(pkt.tcp.seq)
        except:
            continue
        if seq_num in seq:
            # duplicate or retransmission
            pass
        else:
            seq[seq_num] = pkt.sniff_timestamp
        # Simple reorder detection: track expected next seq
PY

For rigorous measurement, embed a monotonically increasing sequence in the payload (e.g., via tcprewrite --inplace --enet-smac=00:11:22:33:44:55 --seed=1) and run a post‑process script that counts out‑of‑order deliveries.

Measuring Netem Performance and Limitations

Benchmarking Netem Emulation

Goal: Quantify how netem’s impairment behaves under a fixed 50 Mbit bandwidth as we vary reorder, gap, and duplicate percentages, and identify the point where netem’s internal mechanics diverge from a realistic lossy link.

Testbed: Two Linux hosts connected via a back‑to‑back 10 GbE NIC (or a programmable switch). Host A runs a traffic generator (iperf3 -c HostB -t 60 -b 50M -P 4). Host B runs the receiver. Netem is placed on Host A’s egress (eth0). All measurements are taken from Host B.

Metrics collected:

• Goodput (Mbps) from iperf3.
• Observed loss (tc -s qdisc show + iperf3 retransmits).
• Duplicate ratio (duplicate packets / total packets) from tcpdump.
• Reorder distance distribution (max gap observed) from a custom sequence‑tracking script.
• CPU utilization (mpstat -P ALL 1) and softirq load.
• Queuing delay (tc -s qdisc show backlog).

Baseline: No netem (just HTB at 50 Mbit). Expect ~50 Mbit goodput, <0.1 % loss, negligible duplicates/reorder.

Identifying Scaling Limitations

Example: Scaling Netem Emulation with Increasing Bandwidth

We repeat the benchmark while raising the HTB ceiling from 10 Mbit to 200 Mbit in 10 Mbit steps, keeping impairment constants at:

reorder 5% gap 10 correlation 0%
duplicate 2%

Interpretation: As the line rate grows, netem’s per‑packet processing becomes a bottleneck. Softirq load climbs, causing the HTB shaper to defer packets, which increases queuing delay and triggers loss in the netem reorder buffer (packets dropped when the internal ring exceeds its limit). The observed duplicate ratio drops because duplicates are discarded by the overloaded shaper before they reach the wire. This demonstrates that netem stops faithfully emulating the requested impairment and starts inventing loss and drop behavior that a real link would not produce at those rates.

Example: Scaling Netem Emulation with Multiple Network Interfaces

We attach identical netem instances to four parallel 10 GbE interfaces, each shaped to 25 Mbit via HTB, and aggregate traffic with iperf3 -P 16. The total offered load is 100 Mbit.

The aggregate goodput scales linearly until the combined softirq load approaches saturation; beyond that, loss and duplicate distortion appear, confirming the single‑interface findings.

Use these procedures to determine the bandwidth envelope within which netem provides a faithful emulation of random reorder, gap, and duplicate impairments, and to recognize when its internal limitations begin to synthesize recovery behavior absent in real production paths.