Proving receiver CPU starvation from packet evidence

Introduction to Network Performance Analysis

Packet timelines give a deterministic view of when bytes leave the sender, travel through the network, and are acknowledged by the receiver. Aligning TCP sequence numbers with timestamps (e.g., tcpdump -tttt -s0 -w capture.pcap) lets us reconstruct each segment’s flight time and spot stalls. In a healthy flow, the inter‑segment gap matches the smoothed round‑trip time (SRTT) and the congestion window (cwnd) grows smoothly. When the receiver advertises a zero‑window, the sender’s transmission rate collapses to zero regardless of cwnd, producing a flat segment in the timeline.

A packet‑timeline graph (time on X‑axis, cumulative bytes sent on Y‑axis) shows:

• Linear ascent while data is in flight.
• Plateau when the sender is blocked by a zero‑window advertisement.
• Sudden jump when the window reopens.

If the plateau coincides with high CPU utilization on the receiver host, the stall may be due to the application not draining the socket buffer; if CPU is idle, the cause is more likely network‑side congestion or a misbehaving middlebox forcing the receiver to advertise zero.

Why Correlate System Metrics?

A packet capture tells us that the sender stopped transmitting, but not why. To distinguish a zero‑window sender symptom from a receiver‑process drain problem we must correlate:

• Socket‑level memory usage (ss -i, /proc/<pid>/fdinfo/<fd>).
• CPU saturation (per‑core utilization, run‑queue length, scheduler latency).
• eBPF‑derived signals (e.g., tcp_retransmit_skb, sched_switch, socket_read latency).
• Control‑plane events (routing changes, QoS policer hits) via gNMI or syslog.

Aligning these timelines lets us assign confidence to a hypothesis. For example, a rise in socket_read_latency eBPF histogram concurrent with a flat packet timeline strongly suggests the receiver application is the bottleneck. Conversely, stable socket read latency with rising NIC drop counters points to network congestion.

Understanding Zero‑Window Sender Symptoms

Definition and Causes

A TCP zero‑window condition occurs when the receiver advertises a receive window (rwnd) of zero in the TCP header, telling the sender to pause transmission until buffer space becomes available. Common causes:

• Application‑side drain latency – the process calls read()/recv() slower than data arrives.
• Socket buffer exhaustion – the SO_RCVBUF limit is reached and the application does not release data.
• Kernel‑level throttling – TCP receive autotuning hitting a ceiling due to memory pressure.
• Middlebox interference – devices that deliberately shrink the advertised window for QoS.
• Receiver‑side packet loss leading to spurious zero‑window probes (rare).

The sender enters the Persist Timer state, sending periodic window probes (usually a single byte) to test for window updates.

Detecting Zero‑Window Advertisements in Packet Captures

Using tcpdump or tshark we can filter for packets where the Window field equals zero and the Acknowledgment Number is advancing (indicating the receiver is still ACKing data):

# Capture only zero‑window advertisements from the receiver side
tcpdump -i any -nn -s 0 -w zero_window.pcap \
  'tcp[14] & 0xF0 >> 2 == 5 and tcp[14+14] == 0'

Explanation: tcp[14] & 0xF0 >> 2 == 5 extracts the data offset (ensuring a valid TCP header); tcp[14+14] == 0 checks the 16‑bit window field at offset 14 (TCP start) + 14 (IP+TCP header) for zero.

In Wireshark, the TCP Zero Window and TCP Window Update expert info flags appear automatically. A timeline of these events shows clusters of zero‑window packets followed by a window update after a persist timeout (typically 0.5–2 s, doubling each probe).

TCP Header Fields Relevant to Zero‑Window Detection

When the window is zero, the sender must not send new data; any non‑zero‑length segment after a zero‑window advertisement is a retransmission or a window probe. Observing a pattern of zero‑window → [Persist Probe] → zero‑window → … → Window Update confirms the classic zero‑window sender symptom.

Correlating System Metrics for Diagnosis

Using ss for Socket Statistics and Memory Data

The ss utility provides instantaneous socket memory usage (rmem_alloc, wmem_alloc) and buffer limits. For a given destination IP/port we can query:

# Show TCP sockets with zero receive window and their memory usage
ss -ti state established '( dport = :80 or dport = :443 )' | \
awk '/rwnd:0/ {print $1, $5, $6, $7}'

Sample output (annotated):

# dst:port   rwnd  recv-q  send-q  rmem_alloc
10.0.2.5:443 0     12288   0       65536

• rwnd:0 confirms zero‑window advertisement from the receiver’s perspective (as seen by the local stack).
• recv-q shows bytes queued but not yet read by the application; a steadily growing recv-q while rwnd stays zero indicates the application is not draining data.

To correlate with a process, map the socket inode to a PID:

# Find the inode for the socket above
INODE=$(ss -ti state established '( dport = :443 )' | \
        awk '/10.0.2.5:443/ {print $NF}' | cut -d'[' -f2 | cut -d']' -f1)
# Locate the process holding this inode
lsof -n -iTCP -sTCP:ESTABLISHED | grep $INODE

If the PID corresponds to a low‑priority worker thread with high voluntary context switches, the drain is likely application‑bound.

Analyzing CPU Saturation

CPU saturation can be observed via:

• Per‑core utilization – mpstat -P ALL 1.
• Run‑queue length – vmstat 1 (column r).
• Scheduler latency – /proc/sched_debug or perf sched latency.

A useful PromQL query (assuming node_exporter) to detect cores consistently >80 % utilized:

# Average utilization per core over the last 5 minutes
avg by (cpu) (rate(node_cpu_seconds_total{mode!="idle"}[5m])) > 0.8

If this alert fires while ss shows growing recv-q and zero rwnd, the receiver CPU is likely the bottleneck. If CPU remains idle (<20 %) yet recv-q grows, the issue may be kernel socket buffer limits or NIC throttling.

Integrating eBPF or Scheduler Signals

eBPF can expose fine‑grained latency of socket reads and scheduler wake‑ups. Example eBPF tracepoint program (BCC) to measure read() latency on a specific PID:

#include <linux/fs.h>
#include <uapi/linux/ptrace.h>
#include <net/sock.h>

struct data_t {
    u32 pid;
    u64 delta_us;
};
BPF_PERF_OUTPUT(events);

TRACEPOINT_PROBE(syscalls, sys_enter_read) {
    u64 pid_tgid = bpf_get_current_pid_tgid();
    u32 pid = pid_tgid >> 32;
    if (pid != 1234) return 0;          // replace 1234 with target PID
    struct data_t data = {};
    data.pid = pid;
    data.delta_us = bpf_ktime_get_ns(); // start timestamp
    bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU, &data, sizeof(data));
    return 0;
}

TRACEPOINT_PROBE(syscalls, sys_exit_read) {
    u64 pid_tgid = bpf_get_current_pid_tgid();
    u32 pid = pid_tgid >> 32;
    if (pid != 1234) return 0;
    u64 *start = start_ts.lookup(&pid);
    if (!start) return 0;
    u64 delta = bpf_ktime_get_ns() - *start;
    events.perf_submit(ctx, &(u64){delta}, sizeof(delta));
    return 0;
}

The resulting histogram of read() latency (e.g., via bpftrace -e 'tracepoint:syscalls:sys_exit_read /pid==1234/ { @latency[args->ret] = hist(nsecs); }') shifts toward higher latencies if the application is slow to consume data.

Scheduler signals can be captured with tracepoint:sched:sched_switch to measure the time a network‑receiving task spends off‑CPU. A rising average off‑CPU time correlates with growing recv-q.

Troubleshooting Zero‑Window Sender Symptoms

Step‑by‑Step Guide

1. Capture the flow

   tcpdump -i any -w flow.pcap host <receiver_ip> and port <port>

2. Detect zero‑window advertisements

   tshark -r flow.pcap -Y "tcp.window_size == 0 && tcp.ack"

3. Correlate with socket state – run ss -ti state established '( dport = :<port> )' at the same timestamps (or via a looping script) to capture rwnd, recv-q, rmem_alloc.

4. Check CPU & scheduler

   mpstat -P ALL 1
   vmstat 1
   # optionally: perf sched record

5. Inspect eBPF latency – attach a read‑latency tracepoint to the receiver PID; observe if latency spikes align with zero‑window periods.

6. Determine root cause

   • If recv-q grows steadily, CPU is low, and read latency is high → application drain issue.
   • If CPU spikes, NIC drop counters rise, and recv-q stays bounded → network congestion or receiver‑side resource pressure.
   • If the window update arrives only after a persist timeout (~0.5‑2 s) and no application latency is seen → middlebox or TCP stack bug.

7. Apply remediation

   • Application – increase thread pool size, enable async I/O, or raise SO_RCVBUF via setsockopt.
   • System – tune /proc/sys/net/core/rmem_max, enable TCP autotuning (net.ipv4.tcp_moderate_rcvbuf), or adjust NIC interrupt coalescence.
   • Network – inspect QoS policers, check for mis‑configured TCP MSS clamping, or replace faulty middlebox.

Using CLI Tools for Troubleshooting

tcpdump snippet (capture only zero‑window packets with timestamps):

tcpdump -i eth0 -nn -s 0 -w zw.pcap 'tcp[14] & 0xF0 >> 2 == 5 and tcp[14+14] == 0' -tttt

The -tttt adds human‑readable timestamps; inspecting the output shows a series of packets with identical timestamps spaced by the persist timer (e.g., 0.5 s, 1 s, 2 s).

Wireshark filter for zero‑window and subsequent window update:

tcp.window_size == 0 || tcp.analysis.window_update

Zero‑window packets appear light blue; window updates appear green. The time delta between a zero‑window and its update reveals the persist interval.

ss loop to monitor socket memory during a zero‑window episode:

while true; do
    ts=$(date +%s.%N)
    ss -ti state established '( dport = :443 )' | grep 'rwnd:0' | awk -v t=$ts '{print t, $0}'
    sleep 0.2
done > zw_ss.log

The log shows lines where rwnd:0 persists while recv-q increments; a flat recv-q with zero rwnd suggests the sender is stalled but the receiver buffer is empty (unlikely; points to a middlebox).

Example Code Snippets for Automated Detection

A simple Python script using scapy to detect zero‑window periods and trigger an alert via systemd-notify:

#!/usr/bin/env python3
from scapy.all import PcapReader, TCP
import time

def detect_zero_window(pcap_file, threshold_sec=2.0):
    last_ts = None
    zero_start = None
    for pkt in PcapReader(pcap_file):
        if TCP in pkt and pkt[TCP].window == 0:
            ts = pkt.time
            if zero_start is None:
                zero_start = ts
            elif ts - zero_start > threshold_sec:
                print(f"ALERT: persistent zero-window from {zero_start:.3f}")
                # Example: systemd-notify --status="Zero-window detected"
                zero_start = None  # reset after alert
        else:
            zero_start = None

if __name__ == "__main__":
    import sys
    if len(sys.argv) != 2:
        print(f"Usage: {sys.argv[0]} <pcap_file>")
        sys.exit(1)
    detect_zero_window(sys.argv[1])

The script prints an alert when a zero‑window condition persists longer than threshold_sec seconds, enabling automated remediation or notification pipelines.