LinkState
Go back

Half-dead ECMP and the flows that suffer

Introduction to Partial Spine Degradation

Definition and Causes

Partial spine degradation occurs when a spine link stays administratively up but shows elevated error rates, increased buffering, or micro‑bursts that affect only a subset of traffic flows. Common causes include:

Effects on Network Performance

Because leaf‑spine fabrics use ECMP with a hash (typically 5‑tuple) to distribute flows, a degraded spine impacts only the flows whose hash selects that spine. Fabric‑wide averages (link utilization, error counters, buffer occupancy) can stay within normal bounds while those flows experience:

Understanding Average Fabric Health Metrics

Overview of Fabric Health Metrics

Fabric health is usually monitored via:

These metrics are aggregated over time and space; they smooth out short‑lived, per‑flow anomalies.

Limitations of Average Metrics in Detecting Partial Spine Degradation

Consequently, relying solely on average fabric health can give a false sense of health while a subset of flows suffers degraded QoS.

Benchmarking Partial Spine Degradation

Experimental Setup

Methodology for Introducing Controlled Degradation

  1. Error injection – Use tc qdisc add dev <spine‑iface> root netem corrupt 0.1% to induce random bit flips (simulating CRC errors) on a single spine link (e.g., S2‑L3). Verify with ethtool -S showing a rise in rx_crc_errors.
  2. Bufferbloat injection – Replace the default queue with a large FIFO and add netem delay 50us limit 10000 to create a standing queue ~ 50 µs deep under load. Observe via tc -s qdisc show rising backlog.
  3. Micro‑burst generation – Superimpose netem loss 0.0% delay 0us distribution normal 10us 5us to create jitter that triggers queue buildup only when multiple flows hash to the same spine.
  4. Validation – Confirm that the link remains up (carrier OK), average utilization stays ~ 30 % (unchanged), but the degraded spine’s error counters or queue depth rise > 10× baseline.

Tools and Techniques for Monitoring and Measurement

ethtool statistics

ethtool -S eth0 | grep -E 'rx_crc_errors|tx_dropped|rx_dropped'

tc queue stats

tc -s qdisc show dev eth0

Output fields: backlog, packets, drops.

eBPF/XDP telemetry

// SPDX-License-Identifier: GPL
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 1<<20);
    __type(key, __u32);
    __type(value, __u64);
} latency SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 1);
    __type(key, __u32);
    __type(value, __u64);
} pkt_cnt SEC(".maps");

SEC("xdp")
int xdp_pass(struct xdp_md *ctx)
{
    __u32 hash = jhash(&ctx->data, ctx->data_end - ctx->data, 0xdeadbeef) & 0xfffff;
    __u64 *ts = bpf_map_lookup_elem(&latency, &hash);
    if (ts) {
        __u64 now = bpf_ktime_get_ns();
        bpf_perf_event_output(ctx, &latency, BPF_F_CURRENT_CPU,
                              &(struct {__u64 delta;}){ .delta = now - *ts },
                              sizeof(struct {__u64 delta;}));
    } else {
        __u64 now = bpf_ktime_get_ns();
        bpf_map_update_elem(&latency, &hash, &now, BPF_ANY);
    }

    __u64 *cnt = bpf_map_lookup_elem(&pkt_cnt, &(const __u32){0});
    if (cnt)
        *cnt += 1;
    return XDP_PASS;
}

char _license[] SEC("license") = "GPL";

Load with:

ip link set dev eth0 xdp obj xdp_latency.o sec xdp

Host‑side TCP stats

ss -i state established '( dport = :5201 )'

Fields of interest: retransmits, rto, rcv_rtt.

Traffic analysis

tcpdump -nn -r capture.pcap -A   # verify UDP sequence gaps
tshark -z io,stat,0,"TCP,tcp.analysis.retransmission" -r capture.pcap

Per‑Flow Loss, Latency, and Retransmit Cost

Impact of Hashed Traffic on Network Performance

ECMP distributes flows based on a deterministic hash. If the hash function is uniform, each spine carries ≈ 1/N of the flow count and byte count. However:

Measuring Per‑Flow Metrics in a Degraded Spine Environment

  1. Flow identification – Use a 5‑tuple hash to steer a known subset of flows to the degraded spine. Example with tc filter:

    # Mark packets with srcIP 10.0.1.0/24, dstIP 10.0.3.0/24, sport 5000-5010
tc filter add dev leaf1-eth0 protocol ip prio 1 u32 \
    match ip src 10.0.1.0/24 \
    match ip dst 10.0.3.0/24 \
    match ip dport 0x1388 0xfffc \
    action skbedit priority 1
tc qdisc add dev leaf1-eth0 handle 1: root prio bands 3
tc filter add dev leaf1-eth0 protocol ip prio 2 u32 \
    match ip src 10.0.1.0/24 \
    match ip dst 10.0.3.0/24 \
    match ip dport 0x1388 0xfffc \
    action skbedit priority 1
tc qdisc add dev leaf1-eth0 parent 1:1 handle 10: netem delay 0us loss 0.0%
tc qdisc add dev leaf1-eth0 parent 1:2 handle 20: netem delay 0us loss 0.0%
tc qdisc add dev leaf1-eth0 parent 1:3 handle 30: netem corrupt 0.1%   # degraded spine path

    This forces matching flows to traverse the degraded spine’s queue (band 3) while others use bands 1‑2.

  2. Latency capture – Enable hardware timestamps on the receiver NIC:

    ethtool -K eth0 tx on rx on
ethtool -K eth0 tx-timestamp on rx-timestamp on
ip link set dev eth0 up

    Then run a receiver that reads SO_TIMESTAMPING:

    int sock = socket(AF_INET, SOCK_DGRAM, 0);
int val = SOF_TIMESTAMPING_TX_HARDWARE | SOF_TIMESTAMPING_RX_HARDWARE |
          SOF_TIMESTAMPING_SOFTWARE;
setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, &val, sizeof(val));
// recvmsg() yields sc->sc_ts with hardware rx timestamp

    Compute delta = rx_timestamp - tx_timestamp (extracted from UDP payload).

  3. Loss detection – Embed a monotonically increasing 32‑bit sequence in each UDP packet; the receiver tracks the expected sequence and logs gaps.

  4. Retransmit cost – For TCP flows, poll ss -i every 200 ms and compute delta retransmits:

    while true; do
    ss -i state established '( dport = :5201 )' | awk '/retransmits/ {print $NF}'
    sleep 0.2
done
Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email
Previous Post
Identity labels churn faster than ACL templates
Next Post
Interrupt coalescing tuning after XDP changes the curve