One packet through a healthy leaf and a sick spine

Overview of Anycast Gateway Ingress

Anycast gateway ingress advertises the same IP prefix (the anycast VIP) from multiple leaf switches. The underlay routing protocol (BGP or OSPF) installs equal‑cost paths to that prefix, allowing the forwarding plane to select a path based on a hash of packet headers. This spreads ingress traffic across the fabric and provides implicit resilience: if one leaf fails, traffic shifts to the remaining leaves without re‑addressing the tenant workload.

At the ingress leaf, the anycast gateway performs the first L3 lookup for the tenant packet. The leaf’s VRF contains the anycast VIP as a local address, so the packet is treated as destined for the local gateway. The gateway then:

Performs a destination MAC lookup (usually the leaf’s own MAC for the anycast IP).
Applies the tenant’s VRF‑specific routing table to determine the next‑hop underlay path toward the remote VTEP.
Invokes the VXLAN encapsulation function (ASIC or kernel vxlan driver) before handing the packet to the fabric.

If the ingress leaf is also a VTEP, the outer UDP header carries the leaf’s VTEP IP as the source address; the destination address is the remote VTEP’s IP, learned via BGP EVPN or static configuration.

Single Tenant Flow Across the Fabric

Initial Packet Processing at Ingress

Consider a TCP SYN from a tenant VM (IP 10.1.2.5, port 54321) to the anycast VIP 10.0.0.1 (port 80). The packet arrives on the leaf’s access port and undergoes:

L2 forwarding – MAC lookup finds the leaf’s anycast MAC; the packet is punted to the L3 forwarding engine.
L3 lookup – In the tenant VRF, destination 10.0.0.1 matches a local anycast address; the packet is considered “to self” and is forwarded to the VXLAN encapsulation stage.

Encapsulation decision – The VNI associated with the tenant’s subnet (e.g., VNI 5010) is retrieved from the VRF‑VNI map. The leaf constructs a VXLAN header:

Outer Ethernet Src MAC = leaf MAC
Outer Ethernet Dst MAC = selected spine MAC
Outer IP Src = leaf VTEP IP (e.g., 10.255.0.10)
Outer IP Dst = spine underlay IP (next hop toward remote VTEP)
Outer UDP Src Port = hash‑derived (0x0000‑0xFFFF)
Outer UDP Dst Port = 4789
VXLAN Flags = I (I=1)
VNI = 5010

Underlay next‑hop resolution – The leaf performs an ECMP hash over the underlay 5‑tuple (src IP, dst IP, src port, dst port, protocol) to select one of the N spine links toward the remote VTEP’s underlay address.

Encapsulation and Forwarding to Spine Layer

After encapsulation, the packet is an Ethernet frame with:

Inner Ethernet – tenant VM MAC → anycast gateway MAC (inner payload unchanged).
IP – inner IP header (tenant VM → anycast VIP).
TCP – SYN, sequence number, etc.
Outer Ethernet – leaf’s MAC → selected spine’s MAC.
Outer IP – src = leaf VTEP IP, dst = spine’s underlay IP.
Outer UDP – src port = hash‑derived, dst port = 4789.
VXLAN – VNI 5010.

The leaf transmits the frame out the selected spine‑facing interface (e.g., Ethernet1/1). The spine strips the outer Ethernet header, performs an IP lookup on the outer destination address (its own underlay IP), and forwards the packet toward the next hop based on the underlay routing table (typically another ECMP set toward the leaf that hosts the remote VTEP).

ECMP Choice and Traffic Distribution

The leaf’s ECMP hash is a symmetric function of the 5‑tuple. For a given flow, the hash yields a deterministic spine index. If all spines are healthy and have equal latency, the flow experiences consistent per‑packet latency and no reordering. The hash is computed in the ASIC pipeline; on many platforms it occupies a few hundred picoseconds and does not cause measurable CPU load.

If the fabric uses UCMP or weighted ECMP, the hash may be biased toward higher‑capacity links, but the same flow sticks to a single spine unless a path change triggers a rehash (e.g., link flap, weight change).

Impact of Degraded Spine on Traffic Flow

Detection of Spine Degradation

A spine that remains link‑up but degraded can manifest as:

Increased interface error counters (CRC, FCS, symbol errors) visible via show interfaces Ethernet1/1 counters errors.
Elevated queue depth and drop counters on the egress queue toward the leaf (show interfaces Ethernet1/1 queue counters).
Increased latency measured by in‑band telemetry (e.g., show latency monitor or external probes).
CPU pressure on the spine’s control plane if it is software‑based (show processes cpu showing high %CPU in the forwarding daemon).
BGP flap damping may not trigger because the link stays up; however, BGP hold‑timer expirations can occur if the spine fails to send keepalives due to CPU overload.

Detection relies on continuous polling of these counters and comparing them to baseline thresholds (e.g., >0.1% error rate, >80% queue occupancy, >2 ms added latency).

Effects on ECMP and Traffic Redistribution

Because the spine stays in the forwarding table, the leaf’s ECMP hash continues to select it for a subset of flows. The degraded spine introduces two primary pathologies:

Latency inflation – packets experience higher serialization and queuing delay, increasing RTT for the flow.
Packet loss – transient microbursts exceed the spine’s buffer capacity, causing tail‑drops. The loss appears as sporadic packet loss in the flow’s statistics.

If degradation is severe enough to cause out‑of‑order delivery (some packets take the degraded path while others take healthy spines), TCP may interpret the reordering as loss and trigger spurious retransmits. The leaf’s ECMP does not react to per‑packet latency; it only reacts to a change in the set of equal‑cost paths (e.g., a spine being withdrawn from the routing table). Therefore, traffic continues to be sent to the degraded spine until an external mechanism (e.g., BGP path‑health, telemetry‑driven route‑map) withdraws or de‑weights the path.

Retransmit Behavior and Packet Loss

Consider the TCP SYN‑ACK flow from the remote VTEP back to the tenant. The return path may also hash to the degraded spine (if the hash is symmetric) or to a healthy spine (if the 5‑tuple differs due to port randomization). Asymmetric loss leads to:

Duplicate ACKs at the sender when a segment is lost, triggering fast retransmit after three dupACKs.
Retransmission timeout (RTO) events if the loss burst exceeds the retransmission window.
Increased RTT variance, causing the TCP congestion controller to raise the congestion window conservatively, reducing throughput.

From the network perspective, the loss appears as an increment in the spine’s egress drop counters and an increment in the leaf’s ingress discard counters (if the leaf receives ICMP Source Quench or similar, though modern networks rarely generate them). The leaf may also see increased out‑of‑order packets on the healthy spines, detectable via show tcp statistics on the end hosts or via telemetry exporting TCP metrics.

Encapsulation and Decapsulation Process

VXLAN Encapsulation at Ingress

At the ingress leaf, VXLAN encapsulation occurs in the ASIC’s lookup‑to‑encap stage:

VNI lookup – VRF → VNI table yields VNI 5010.
UDP port calculation – Many implementations use a hash of the inner 5‑tuple to distribute inner flows across UDP source ports (to avoid inner‑flow polarization). The resulting UDP src port may be, for example, 0x1A2B.
Header construction – Outer MAC (leaf → spine), outer IP (leaf VTEP → spine underlay), outer UDP (src port = calculated, dst port = 4789), VXLAN header (I=1, VNI=5010, reserved=0).
Checksum offload – If the NIC supports UDP checksum offload, the outer UDP checksum is computed by the NIC; otherwise, the ASIC calculates it.

The encapsulated packet is then handed to the forwarding stage for ECMP spine selection.

VLAN Tagging and VNI Mapping

In many designs, the tenant VLAN is mapped to a VNI locally at the leaf (VLAN‑to‑VNI translation). The ingress port may be configured as a trunk; the packet arrives tagged with VLAN 101. The leaf’s VLAN‑VNI table maps VLAN 101 → VNI 5010. After 802.1Q de‑tagging, the inner Ethernet header is exposed for encapsulation. This mapping is typically stored in TCAM; a miss results in packet drop and a log entry (%VLAN-VNI-MAP-MISS).

Decapsulation at Remote VTEP

At the remote leaf (the VTEP that owns the tenant’s destination MAC), the process reverses:

Outer header validation – Verify VXLAN I flag, VNI matches a locally configured VNI for the tenant VRF.
Checksum verification – If UDP checksum offload is enabled, the NIC verifies it; otherwise, the ASIC does.
Decap – Strip outer Ethernet, IP, UDP, and VXLAN headers, exposing the inner Ethernet frame.
VNI‑to‑VLAN mapping – Look up VNI 5010 → VLAN 101, re‑tag the inner frame with VLAN 101.
L2 forwarding – Perform MAC lookup for the destination tenant VM and forward out the appropriate access port.

If the remote VTEP detects a VNI mismatch or invalid checksum, it increments the vxlan decap error counter and silently drops the packet (no ICMP is sent to avoid amplification).

Troubleshooting Degraded Spine Scenarios

Identifying Packet Loss and Jitter

Loss – Use show interfaces Ethernet1/1 counters on the spine to inspect in discards, out discards, and in errors. Correlate with show logging for any LINK-ERROR messages.
Jitter/latency – Enable in‑band telemetry (monitor session 1 type erspan-source) or use ping/traceroute with timestamp options from a host to measure RTT variance. On the spine, show latency monitor (if available) provides per‑queue latency histograms.
CPU – show processes cpu sorted to see if the forwarding daemon (l2ribd, fibd) is consuming >70% CPU, which can cause increased processing delay and packet drops.

Analyzing ECMP Table and Routing Decisions

ECMP hash verification – On the leaf, show platform software fed switch active (or the platform‑specific equivalent) displays the current ECMP groups and the hash seeds used.
Routing table – show ip route vrf <tenant-vrf> 10.0.0.1 confirms the anycast VIP is installed as a local address.
Path health – If telemetry‑driven route‑maps are in place, show route-map and show policy-map reveal whether the degraded spine has been de‑weighted or withdrawn.

By correlating interface error counters, latency histograms, and ECMP hash outputs, administrators can pinpoint whether a degraded spine is causing loss, latency inflation, or spurious TCP retransmits, and apply corrective actions such as adjusting link weights, triggering BGP path‑withdrawal, or redistributing traffic via UCMP.