Compare per‑flow retransmits caused by one bad ECMP next hop with symptoms caused by fabric‑wide congestion and show how to prove which story fits

By Nadia Kareem – Routing Architect & Control‑Plane Specialist

Introduction to ECMP and Congestion

Overview of ECMP

Equal‑Cost Multi‑Path (ECMP) installs multiple next‑hop entries for a destination prefix when the routing protocol reports several paths with identical metric. The control‑plane (RIB) holds a list of equal‑cost next‑hops; the data‑plane (FIB) programs a hash‑based load‑balancing table that selects one of those next‑hops per flow (or per packet, depending on the platform).

The hash is stable for a given flow as long as the input fields remain unchanged; therefore a per‑flow ECMP path persists until a topology change (link flap, metric change) or a re‑hash event (e.g., ECMP member addition/removal, hash‑seed change).

If one ECMP next‑hop is bad (faulty NIC, mis‑configured QoS, high bit‑error link), only the flows that hash to that NH experience packet loss, latency spikes, or jitter. TCP sees those losses as retransmits, while other flows remain unaffected.

Understanding Fabric‑Wide Congestion

Fabric‑wide congestion occurs when the aggregate offered load exceeds the capacity of one or more internal links across the entire switching fabric (e.g., a Clos spine‑leaf fabric, data‑center fabric, or MPLS core). Unlike a single bad ECMP NH, congestion is not tied to a specific next‑hop; it manifests as increased queuing delay and random packet drops that affect all traffic traversing the congested element, regardless of flow hash.

Control‑plane state remains unchanged: the RIB still lists the same set of equal‑cost NHs, and the FIB still programs the same ECMP hash table. The difference lies in the data‑plane: the forwarding ASIC experiences buffer occupancy beyond its threshold, triggering tail‑drop or explicit congestion notification (ECN) marks.

Typical observable symptoms:

• Elevated interface output drop counters on all uplinks/downlinks of the affected fabric element.
• Increased latency and jitter measured via ping/tracepath across multiple destinations.
• Uniform rise in TCP retransmit rates across many flows, often correlating with time‑of‑day traffic bursts.

Symptoms of Per‑Flow Retransmits

Identifying Retransmit Issues

Retransmits are visible at the transport layer (TCP) and can be probed via:

• Host‑side: netstat -s (Linux), netstat -p tcp (Windows), ss -s (Linux). Look for retransmits, fastretrans, timeoutretrans.
• Network‑side: Flow‑export (IPFIX, sFlow, NetFlow) fields tcpRetransmitPkts or tcpRetransmitBytes.
• Packet capture: TCP duplicate ACKs (Dup ACK) followed by retransmission, or retransmission timeout (RTO) events.

When only a subset of flows shows retransmits, the pattern often aligns with a specific ECMP next‑hop.

Distinguishing Between ECMP and Congestion Symptoms

Troubleshooting Per‑Flow Retransmits

Using CLI Commands for Troubleshooting

Below are representative commands for three common network operating systems. Adjust syntax for your vendor/release.

Cisco IOS‑XR

# 1. Verify ECMP composition for the suspect prefix
show route 10.0.0.0/24 detail
# Output shows: 2 paths, via 10.1.1.2 (Gig0/0/0/1) and 10.1.1.3 (Gig0/0/0/2)

# 2. Check per‑interface output drops (candidate bad NH)
show interfaces Gig0/0/0/1 | include output drops
show interfaces Gig0/0/0/2 | include output drops

# 3. Examine ECMP hash bucket distribution (if platform supports)
show cef 10.0.0.0/24 detail | include load sharing
# Look for bucket counts per next‑hop

# 4. Monitor TCP retransmits from a host (via SNMP or telemetry)
show snmp mib tcpMIB tcpRetransSegs

Juniper Junos

# 1. ECMP next‑hop list
show route 10.0.0.0/24 extensive | match "^(Static|OSPF|BGP)"
# Shows: 2 next hops: 10.1.1.2 via ge-0/0/1.0, 10.1.1.3 via ge-0/0/2.0

# 2. Interface error counters
show interfaces ge-0/0/1.0 extensive | match "output errors|output drops"
show interfaces ge-0/0/2.0 extensive | match "output errors|output drops"

# 3. ECMP hash statistics (if supported)
show route forwarding-table destination 10.0.0.0/24 detail | match "load-balancing"

Arista EOS

# 1. ECMP routes
show ip route 10.0.0.0/24 vrf default
# Shows: 2 paths via Ethernet1 and Ethernet2

# 2. Interface counters
show interfaces Ethernet1 | include output drops
show interfaces Ethernet2 | include output drops

# 3. ECMP load‑balancing info (platform‑dependent)
show hardware counter | include ecmp

Interpretation:
If only one member interface shows a rising output‑drop counter (or increased CRC/frame errors) while its sibling stays clean, the bad NH hypothesis gains weight.

Analyzing Packet Capture Data

1. Capture at the ingress of the ECMP group (e.g., on the leaf switch facing the servers).

2. Apply a capture filter that isolates the problematic TCP flow (e.g., tcp and src host 10.0.1.10 and dst host 10.0.2.20).

3. In Wireshark/tcpdump:

   • Look for duplicate ACKs ([TCP Dup ACK]) followed by a fast retransmit.
   • Measure the round‑trip time (RTT) before loss; a sudden RTT jump often points to a single hop queuing.
   • Use the TCP expert info to see if retransmits are triggered by timeout (RTO) vs fast retransmit. Timeout‑based retransmits suggest longer‑lasting loss (e.g., a flaky link).

4. Correlate with ECMP hash:

   • Extract the 5‑tuple (srcIP, dstIP, srcPort, dstPort, proto) of the retransmitting flow.
   • Compute the hash used by the platform (many vendors publish the hash function; e.g., Cisco uses a CRC‑16 on the 5‑tuple).
   • Verify that the hash maps to the suspect NH.

If many different flows (different 5‑tuples) all hash to the same NH and show retransmits, the bad NH story is consistent.

Identifying Bad ECMP Next Hops

Fabric‑Wide Congestion Troubleshooting

Monitoring Fabric Utilization

Fabric utilization is best observed via counter‑based telemetry on each fabric link (spine‑leaf uplink/downlink, inter‑spine links, etc.).

• SNMP: ifInOctets, ifOutOctets, ifInUcastPkts, ifOutUcastPkts, ifOutDiscards.
• gNMI/Telemetry: openconfig**: interfaces interface[name=] state counters in-discards, out-discards, queue [name=] depth`.
• sFlow: Provides per‑packet sampling and counter polling; can derive utilization and detect microbursts.

A sudden rise in output discards across all members of an ECMP group, coupled with increased queue depth, signals fabric congestion.

Detecting Congestion Hotspots

1. Aggregate per‑link utilization:

   # Example using Prometheus query (if exporting via gNMI)
   sum(rate(interface_out_octets[5m])) by (device, interface)

   Look for any link > 80 % sustained utilization.

2. Microburst detection:

   • Enable burst‑statistics on ASICs (e.g., Cisco Nexus show hardware rate-limit).
   • Look for short‑term queue spikes (e.g., > 95 % depth for < 10 ms) that correlate with retransmits.

3. ECN/CE marking:

   • If ECN is enabled, monitor ecnCeMarks counters. A rise in CE marks across many links indicates congestion before packet drop.

Analyzing Traffic Patterns

• Use flow‑export (IPFIX/sFlow) to break down traffic by:

  • Source/destination prefix
  • Application (port)
  • DSCP/ECN field

• Plot bytes per second per prefix over time. If a single prefix or a small set of prefixes drives > 60 % of the fabric load, congestion may be traffic‑biased rather than truly fabric‑wide.

• Conversely, if all prefixes show a proportional increase in load and the fabric links are uniformly saturated, the congestion is truly fabric‑wide.

Code Examples for Troubleshooting

CLI Commands for ECMP and Congestion Analysis

Below is a vendor‑agnostic Bash snippet that can be run on a device with SSH access (adjust command strings per OS). It gathers ECMP composition, per‑interface drop counters, and computes a simple “bad‑NH score”.

#!/usr/bin/env bash
# Usage: ./ecmp_check.sh <prefix> [vrf]
PREFIX=$1
VRF=${2:-default}

# 1. Get ECMP next-hops (Junos example; replace with appropriate CMD)
if [[ "$(uname -s)" == "Linux" ]]; then
    # Assume we are on a Linux host with vtysh
    CMD="vtysh -c \"show route $PREFIX\""
else
    CMD="show route $PREFIX"
fi

echo "=== ECMP composition for $PREFIX (vrf $VRF) ==="
eval $CMD | grep -A 20 "nexthop"

# 2. Check per‑interface output drops (example for Cisco IOS‑XR)
echo "=== Output drops per interface ==="
for IF in $(eval $CMD | awk '/via/ {print $NF}' | tr -d '(),'); do
    echo -n "$IF: "
    show interfaces $IF | grep "output drops" || echo "counter not found"
done

Replace the show route and show interfaces calls with the appropriate commands for your platform (Junos, EOS, etc.).

End of document.