Network Performance Issues: Isolating Receive‑Queue Imbalance, CPU Starvation, and True Ingress Loss

Introduction

A five‑tuple (source IP, destination IP, source port, destination port, protocol) uniquely identifies a flow in the Linux networking stack. When only a subset of flows shows jitter or retransmissions while the rest of the traffic appears healthy, the problem is usually flow‑localized rather than a global datapath failure. Load‑dependent effects such as RSS/RPS hashing, interrupt affinity, or per‑CPU softirq throttling can steer certain five‑tuples onto a congested receive queue or a starved CPU, leaving other flows on less‑contended paths.

Under increasing ingress load the kernel must:

1. Steer packets to a receive queue (RSS/RPS) based on a hash of the five‑tuple.
2. Raise a NET_RX softirq on the CPU that owns the queue.
3. Process the packet through the protocol stack (IP → TCP/UDP → socket).
4. Deliver data to the application via the socket’s receive queue.

If any step becomes a bottleneck, latency, jitter, and loss appear only for the flows that hash to the affected queue/CPU. Aggregate counters (e.g., ifconfig RX packets) may still look healthy, while a subset of five‑tuples exhibits increased RTT variance and retransmission timeout (RTO) triggers.

Identifying Receive‑Queue Imbalance

Symptoms

• Per‑queue RX drop counters rise on one or a few queues while others stay near zero.
• /proc/interrupts shows a disproportionate number of NET_RX interrupts on a specific CPU.
• Application‑level latency histograms reveal a bimodal distribution: a fast mode for most flows and a slow tail for the affected five‑tuples.
• ethtool -S reports increasing rx_queue_0_drop, rx_queue_1_drop, … with a skewed distribution.

Detection Commands

# Show per‑queue RX packets and drops
ethtool -S eth0 | grep -E 'rx_queue_[0-9]+_packets|rx_queue_[0-9]+_drop'

# Show current RSS indirection table (which CPU each hash maps to)
ethtool -x eth0

# Show interrupt affinity for the NIC
cat /proc/interrupts | grep eth0

If queues 0 and 1 have vastly different packet/drop counts, the NIC’s RSS hash is likely sending a hot set of five‑tuples to those queues.

Example Monitoring Script (Python + pyroute2)

#!/usr/bin/env python3
import time, os, glob
from pyroute2 import IPRoute

def get_queue_stats(ifname):
    base = f"/sys/class/net/{ifname}/queues/rx-"
    stats = {}
    for qdir in sorted(glob.glob(base + "*")):
        qnum = os.path.basename(qdir)
        try:
            rx_packets = int(open(os.path.join(qdir, "rx_packets")).read().strip())
            rx_drop    = int(open(os.path.join(qdir, "rx_drop")).read().strip())
            stats[qnum] = (rx_packets, rx_drop)
        except FileNotFoundError:
            continue
    return stats

if __name__ == "__main__":
    iface = "eth0"
    prev = {}
    while True:
        cur = get_queue_stats(iface)
        for q, (pkt, drop) in cur.items():
            p_prev, d_prev = prev.get(q, (0, 0))
            delta_p = pkt - p_prev
            delta_d = drop - d_prev
            if delta_p:
                loss_ratio = delta_d / delta_p
                print(f"{time.strftime('%H:%M:%S')} {iface}-{q}: Δpkt={delta_p:,} Δdrop={delta_d:,} loss%={loss_ratio*100:.3f}")
        prev = cur
        time.sleep(1)

The script prints per‑queue packet and drop deltas; a sustained high loss% on a single queue points to RSS imbalance.

Troubleshooting CPU Starvation

Symptoms

• Softirq backlog grows on a specific CPU (/proc/softirqs shows rising NET_RX count without proportional processing).
• ksoftirqd CPU usage spikes (top shows high %si on one core).
• Application threads pinned to the same core experience increased latency, even though NIC queues are not dropping.
• perf top reveals a large share of cycles spent in __netif_receive_skb_core or process_backlog on the affected CPU.

Detection Commands

# Per‑CPU softirq statistics
watch -n 1 "cat /proc/softirqs | grep NET_RX"

# Per‑CPU utilization (including softirq)
mpstat -P ALL 1

# Map each queue’s interrupt to its CPU
cat /proc/interrupts | grep eth0 | awk '{print $1,$NF}'

If NET_RX on CPU 2 climbs steadily while other CPUs stay flat, the NIC is directing interrupts to CPU 2 but the softirq handler cannot keep up.

Example bpftrace One‑Liner

sudo bpftrace -e '
tracepoint:softirq:net_rx_entry { @start[cpu] = nsecs; }
tracepoint:softirq:net_rx_exit  { @lat[cpu] = hist(nsecs - @start[cpu]); }
'

The resulting histogram shows tail latency > 10 µs on a particular CPU, indicating starvation.

Diagnosing Actual Ingress Loss

Understanding Ingress Loss

True ingress loss occurs when packets are dropped before they reach the socket receive queue. Common causes:

• RX ring overflow (NIC’s internal descriptors exhausted) → hardware drop.
• Driver‑level drops due to missing buffers (rx_no_buffer_count).
• XDP/eBPF drop actions (if a program is attached).
• LRO/GRO misbehavior causing packet corruption and subsequent discard.

Unlike queue imbalance or CPU starvation, ingress loss reduces the total packet count seen by the stack (/proc/net/dev RX packets) and increments drop counters that are not per‑queue (e.g., rx_errors).

Detection Techniques

# Capture on the NIC and monitor kernel drop trace
sudo tcpdump -i eth0 -nn -s 0 -w /tmp/cap.pcap &
sudo cat /sys/kernel/debug/tracing/trace_pipe | grep -E "netdev_drop|xdp_drop"

If tcpdump sees fewer packets than the traffic generator sends and the kernel drop trace shows netdev_drop events, loss is happening at the NIC/driver layer.

Example Loss‑Detection Script (Python + pyshark)

#!/usr/bin/env python3
import pyshark

def count_pcap(pcap_file, filter_str):
    cap = pyshark.FileCapture(pcap_file, display_filter=filter_str)
    cnt = sum(1 for _ in cap)
    cap.close()
    return cnt

if __name__ == "__main__":
    nic_cap   = "/tmp/nic.pcap"
    pre_cap   = "/tmp/pre_nic.pcap"
    fivetuple = "ip.src==10.0.0.5 && ip.dst==10.0.0.6 && tcp.srcport==12345 && tcp.dstport==80"
    nic_cnt   = count_pcap(nic_cap, fivetuple)
    pre_cnt   = count_pcap(pre_cap, fivetuple)
    loss      = pre_cnt - nic_cnt
    print(f"Five-tuple {fivetuple}: sent={pre_cnt:,} received={nic_cnt:,} lost={loss:,} ({loss/pre_cnt*100:.2f}%)")

A non‑zero loss indicates true ingress loss for that flow.

Analyzing Jitter and Retransmissions

Understanding Jitter and Retransmissions

• Jitter (variation in one‑way delay) often stems from queueing delay in the NIC’s RX ring or the kernel’s backlog when a CPU cannot drain packets fast enough.
• Retransmissions are triggered when the TCP sender’s RTO expires, which can be caused by actual loss or spurious timeout due to excessive RTT variance (jitter).

When only certain five‑tuples show jitter/retransmits, the root cause is usually per‑queue or per‑CPU latency rather than global loss.

Analysis Commands

# Measure one‑way delay with iperf3 (reverse mode) and export JSON
iperf3 -c server -R -i 1 -J --logfile /tmp/iperf.json

# Extract jitter from JSON
jq '.intervals[].sum.jitter_ms' /tmp/iperf.json

# Compute RTT variance with tcptrace
tcptrace -l /tmp/cap.pcap

If jitter spikes coincide with high rx_queue_X_drop or rising NET_RX softirq on a specific CPU, the delay is queue‑induced.

Scaling Limitations and Considerations

Limitations

• RSS hash collisions: many five‑tuples map to the same RX queue, overloading that queue’s CPU.
• Interrupt throttling: NICs may coalesce interrupts; under microburst load the coalescing timer can cause bursts of packets to arrive when the CPU is still processing the previous batch, leading to temporary backlog.
• Softirq rate limit: the kernel caps softirq processing to prevent livelock; if the NIC generates interrupts faster than this cap, packets begin to drop in the NIC’s RX ring (hardware loss).

Mitigation Strategies

Load‑Balancing Techniques

• Increase RSS indirection table size: ethtool -L eth0 combined 4 (or higher).
• Pin interrupts to distinct CPUs via /proc/irq/*/smp_affinity.
• Use RPS/RFS to spread packet processing across CPUs even when the NIC has fewer queues than cores.
• Enable flow‑directional load balancing (e.g., ethtool -K eth0 ntuple on and configure ethtool -N eth0 flow-type tcp4 src-ip … action 2).

Example Configuration for Scalable Queues

# Allocate 8 RX and 8 TX queues
ethtool -L eth0 combined 8

# Verify RSS indirection (should show round‑robin across CPUs 0‑7)
ethtool -x eth0

# Bind each queue’s interrupt to a dedicated CPU
for i in {0..7}; do
    IRQ=$(grep -i eth0 /proc/interrupts | awk -v q=$i '$0 ~ "eth0-"q {print $1}')
    echo $((1<<i)) > /proc/irq/$IRQ/smp_affinity
done

# Enable RPS (distribute packets to CPUs 0‑7)
echo ffffffff > /sys/class/net/eth0/queues/rx-0/rps_cpus
# Repeat for rx-1 … rx-7 or set a mask covering all desired CPUs

These steps help distribute traffic evenly, reduce per‑queue imbalance, alleviate CPU starvation, and minimize true ingress loss under load.

By systematically checking queue balance, softirq processing, and ingress drop counters, you can pinpoint whether observed jitter, retransmissions, or loss stem from receive‑queue imbalance, CPU starvation, or genuine packet drops, and apply the appropriate tuning or scaling measures.