Intended SLO signal versus packet truth

Introduction to Network Degradation Analysis

Latency health is usually expressed through time‑series metrics derived from device counters or host telemetry—interface latency, RTT, jitter, and packet loss. These aggregates (e.g., 95th‑percentile) are visualized on dashboards to convey overall health.

When a short, user‑visible degradation occurs, dashboard averages can mask the underlying per‑packet behavior. Packet captures (PCAP) and retransmit timelines provide the granular view needed to pinpoint the cause.

Understanding Dashboard Claims

Interpreting Latency Health Indicators

A typical latency‑health panel might use a PromQL query like:

# 95th percentile one‑way latency (ms) over the last 5m, interface eth0
histogram_quantile(0.95,
  sum by (le) (
    rate(interface_latency_seconds_bucket{iface="eth0"}[5m])
  )
) * 1000

What the dashboard proves – the statistical distribution of latency samples exported by the device or host over the query window.
What it cannot prove – the exact cause of any outlier, per‑flow behavior, or whether the measured latency matches the user‑experienced path.

Limitations of Dashboard Metrics

1. Aggregation loss – percentiles hide the tail; a brief spike affecting <1 % of packets may be invisible.
2. Sampling interval – counters polled every 30 s can average out a 2‑second degradation.
3. Clock synchronization – one‑way latency relies on NTP/PTP; drift introduces systematic error.
4. Missing context – no visibility into queue depth, buffer occupancy, or hardware timestamps.
5. Label cardinality – high‑cardinality labels (e.g., per‑flow) are often dropped, preventing drill‑down to the offending flow.

Packet Capture Analysis

Configuring Packet Capture Tools

Capture as close as possible to the symptom (client, server, or intermediate switch).

Linux host with tcpdump

# Capture TCP/UDP on eth0, 96‑byte snaplen, rotate 100 MB files, gzip compression
sudo tcpdump -i eth0 -s 96 -w /var/log/pcaps/latency_deg.pcap \
    -C 100 -W 5 -z gzip \
    'tcp[13] & 0x07 != 0'   # capture packets with SYN, FIN, or RST flags

• Hardware timestamps – add --time-stamp-type=adapter if the NIC supports it.
• Buffer size – increase with -B 4096 to avoid drops during bursts.

Switch/router via gNMI

gnmi_cli -addr router.example.com:9339 -username admin -password \
    -alsologtostderr -timeout 10s \
    -subscribe -encode json -path \
    '/network-instance[name=default]/protocols/protocol[name=bgp]/packet-capture' \
    -mode stream -output /tmp/switch_capture.pcap

Analyzing Packet Capture Data for Latency Issues

Using Wireshark or tshark:

# TCP RTT for flow 10.0.0.5:54321 → 10.0.1.10:80 (stream 42)
tshark -r latency_deg.pcap -Y "tcp.stream eq 42" \
    -T fields -e frame.time_relative -e tcp.analysis.ack_rtt \
    > flow42_rtt.tsv

• Latency – tcp.analysis.ack_rtt gives measured RTT per ACK.
• Out‑of‑order – tcp.analysis.out_of_order flags early arrivals.
• Zero‑window – tcp.analysis.zero_window indicates receiver buffer exhaustion.

Identifying Retransmit Patterns and Trends

Retransmits appear as duplicate ACKs or timeouts:

tshark -r latency_deg.pcap -Y "tcp.analysis.retransmission" \
    -T fields -e frame.time_relative -e tcp.seq -e tcp.ack \
    > retransmits.tsv

Retransmit rate per second (awk)

awk '{int=int($1); count[int]++} END{for(i in count) print i, count[i]}' \
    retransmits.tsv > retransmit_rate.csv

A burst of retransmits within a ~200 ms window often aligns with a latency spike visible in the dashboard’s tail latency.

Retransmit Timeline Analysis

Understanding Retransmit Timelines and Their Impact on Latency

A retransmit timeline is an ordered list of events:

Each retransmit adds at least one RTT to the flow’s completion time. For latency‑sensitive applications (SSH, VoIP), even a single retransmit can push perceived latency above a usability threshold.

Correlating Retransmit Timelines with Packet Capture Data

Join retransmit events with nearby packets (pseudo‑SQL):

SELECT r.timestamp AS retransmit_time,
       p.frame_time AS packet_time,
       (p.frame_time - r.timestamp) AS delta_s,
       p.tcp.analysis.ack_rtt AS rtt_ms
FROM retransmits r
JOIN packets p
  ON p.frame_time BETWEEN r.timestamp - 0.05 AND r.timestamp + 0.05
WHERE ABS(p.frame_time - r.timestamp) < 0.05;

• Small delta (<5 ms) – retransmit triggered by a packet captured just before loss.
• Large delta – delayed loss detection (e.g., large RTO).
• Latency impact – sum of RTT values for all retransmits approximates extra delay contributed by loss.

Identifying Discrepancies Between Dashboard Claims and Actual Behavior

• Dashboard claim – 95th‑percentile latency = 12 ms (stable).
• Retransmit timeline – three retransmits in a 150 ms window, each RTT ≈ 30 ms → ~90 ms extra delay for affected flows.
• Packet capture – retransmits belong to a single long‑lived TCP flow experiencing micro‑bursts of queue overflow on an intermediate switch.

The dashboard’s aggregate missed the micro‑burst because it affected <0.5 % of packets and was smoothed over the polling interval.

Troubleshooting Network Degradation

Common Causes of Latency Discrepancies

CLI Commands for In‑Depth Analysis

Interface queue depth (Linux)

# Current transmit queue length
cat /sys/class/net/eth0/statistics/tx_queue_len

# Monitor drops/overruns every 0.5 s
watch -n 0.5 "cat /proc/net/dev | grep eth0"

TCP statistics

ss -s   # summary of TCP states, retransmits, timeouts
netstat -st | grep -E "retrans|timeout"

Hardware timestamp verification

ethtool -T eth0   # shows timestamping capabilities (hardware, software)

gNMI telemetry snapshot (queue depth)

gnmi_cli -addr router.example.com:9339 -username admin -password \
    -alsologtostderr -timeout 5s \
    -path '/components/component[name=FPC0]/queue/queue[name=0]/depth' \
    -encode json

Example Code Snippets

Python – correlate retransmits with latency spikes

import pandas as pd

retrans = pd.read_csv('retransmits.tsv', sep='\t',
                      names=['time', 'seq', 'ack'])
latency = pd.read_csv('flow42_rtt.tsv', sep='\t',
                      names=['time', 'rtt_ms'])

# Mark windows ±0.05 s around each retransmit
retrans['window_start'] = retrans['time'] - 0.05
retrans['window_end']   = retrans['time'] + 0.05

def is_retransmit(t):
    return ((retrans['window_start'] <= t) &
            (retrans['window_end']   >= t)).any()

latency['retransmit_window'] = latency['time'].apply(is_retransmit)
print(latency.loc[latency['retransmit_window'], 'rtt_ms'].describe())

Bash – detect micro‑bursts via drop counters

while true; do
  drops_before=$(cat /sys/class/net/eth0/statistics/rx_drops)
  sleep 1
  drops_after=$(cat /sys/class/net/eth0/statistics/rx_drops)
  echo "$(date +%T) drops in last second: $((drops_after - drops_before))"
done

Scaling Limitations and Considerations

Impact of Network Scale on Latency Health Metrics

• Series explosion – per‑interface counters scale linearly with port count; per‑flow metrics would grow O(N·F).
• Polling interval trade‑off – to keep scrape latency low at scale, intervals are often increased, reducing temporal resolution for short events.
• Storage cost – high‑resolution histograms (exponential buckets) consume significant TSDB space when retained long‑term.

Limitations of Packet Capture and Retransmit Timeline Analysis at Scale

1. Capture bandwidth – full‑packet capture on a 100 Gbps uplink needs >12 GB/s storage; impractical for continuous monitoring.
2. Processing overhead – real‑time reassembly and retransmit detection adds CPU load on capture hosts.
3. Span limits – SPAN/ERSPAN sessions may drop packets under high load, biasing observed retransmit rates.
4. Legal/privacy – capturing payloads may violate policies; often limited to headers only.

Strategies for Overcoming Scaling Limitations

• Selective triggering – start a limited‑duration PCAP buffer when dashboard anomalies exceed a threshold (e.g., tcpdump -W 1 -C 50 -G 30).
• Hardware‑assisted tracing – use switch ASIC telemetry (Intel Tofino register reads, Broadcom counter snapshots) to capture per‑queue latency and drop counts without host‑CPU overhead.
• Flow sampling – employ sFlow or IPFIX with probabilistic sampling (1 in 1000) to retain statistical visibility of retransmits while limiting volume.
• Edge‑only capture – restrict PCAP to edge devices (client/server) where the symptom appears, avoiding core‑network capture.
• Telemetry compression – export retransmit events as compact protobuf messages (retransmit_event { seq, timestamp, cause }) instead of full PCAP.

Case Study: Real‑World Example of Latency Health Discrepancy

Overview of the Case Study Network Environment

• Topology – Two‑tier leaf‑spine data center; 48‑port 25 GbE leaf switches, 4‑spine 100 GbE switches.
• Hosts – Linux servers with Mellanox ConnectX‑6 NICs, hardware timestamping enabled.
• Monitoring – Prometheus scraping interface latency histograms every 15 s; gNMI streaming of queue depth and drop counters every 5 s.
• Symptom – Users reported intermittent SSH latency spikes of 200‑500 ms lasting ~2 s, occurring roughly every 20 min.

Analysis of Packet Capture and Retransmit Timeline Data

• Triggered capture – upon detecting 95th‑percentile latency > 100 ms, a 30‑second PCAP was spawned on the affected leaf switch’s CPU port via ERSPAN to a capture server.
• Findings
  • PCAP showed a micro‑burst of ~15 kB (≈ 5 ms at 25 Gbps) exceeding the egress queue depth of 64 KB on the leaf’s uplink to the spine.
  • Retransmit timeline extracted from the capture: three retransmits within a 12 ms window.

The dashboard’s aggregate latency remained stable because the micro‑burst affected a tiny fraction of packets and was smoothed over the 15‑second scrape interval. Packet capture and retransmit analysis revealed the true cause: brief queue overflow on the leaf‑to‑spine link.

End of document.