Skip to content
LinkState
Go back

Constrain gateway scope or debug forever

Introduction

Stretched Layer‑2/Layer‑3 fabrics (EVPN‑VXLAN, MPLS‑based DCI) need a default‑gateway address reachable from any leaf where a workload may reside. Two common designs are:

  1. Universal leaf‑local anycast gateway – the same anycast IP (and MAC, if L2) is configured on every leaf.
  2. Per‑domain gateway placement – distinct gateway IPs are assigned to smaller failure domains (rack, pod, aggregation block); only leaves in that domain advertise the address.

Both provide first‑hop routing while preserving mobility, but they differ in control‑plane state, failure domains, and operational model. This memo compares them across mobility scope, blast radius, operational clarity, troubleshooting, and provability, then offers guidance for selecting the design that matches operator intent.


Mobility Scope

Universal Leaf‑Local Anycast

Per‑Domain Gateway

Implications

AspectUniversal Leaf‑Local AnycastPer‑Domain Gateway
Maximum mobilityFabric‑wide (any leaf in VNI/VRF)Limited to domain boundary
Host‑side state change on moveOnly MAC binding changes; IP unchangedDefault‑gateway IP may change
Need for host mobility protocolGenerally noneOften required for seamless cross‑domain move
ARP/ND impactLocal MAC update; no ARP refresh neededNew ARP/ND resolution for new gateway IP
Operational assumptionWorkloads retain same IP/gateway across fabricWorkloads assumed static within a domain

Blast Radius

Universal Leaf‑Local Anycast

Per‑Domain Gateway

Implications

AspectUniversal Leaf‑Local AnycastPer‑Domain Gateway
Single leaf failureNo gateway loss; traffic rebalancedNo impact unless leaf is last in its domain
Domain‑wide failure (rack power loss)Gateway still reachable via other leavesGateway lost for that domain; hosts lose default gateway
Correlated failuresRequires simultaneous loss of all leaves to break gatewayAny failure that empties a domain breaks gateway for that domain
Recovery mechanismAutomatic via routing reconvergence; no host actionMay require ARP/ND refresh, DHCP renewal, or mobility protocol
Blast‑radius predictabilityUniform across fabric; depends on number of leaves advertising prefixHeterogeneous; each domain has its own radius

Operational Clarity

Universal Leaf‑Local Anycast

Per‑Domain Gateway

Implications

AspectUniversal Leaf‑Local AnycastPer‑Domain Gateway
Distinct gateway prefixes1 per VNI/VRFN (one per domain)
ECMP next‑hop countEqual to number of leaves advertising prefix (often large)Usually 1 per domain (if single leaf) or small set
Policy scopeSingle prefix → simpleMultiple prefixes → per‑domain policies
Drift detectionVerify uniform presence/absenceVerify per‑domain presence and cross‑domain absence
Proof of reachabilityShow ECMP set in every leafShow prefix present only in intended domain’s leaves
Operational overheadLow (single address)Higher (address book, mapping, redundancy protocols)

Troubleshooting

Universal Leaf‑Local Anycast

  1. Symptom – Host cannot reach default gateway (ping fails).
  2. First check – Verify host’s ARP/ND entry for the gateway IP resolves to a MAC belonging to a leaf (show arp / show ipv6 neighbors).
  3. Next‑hop validation – On the leaf the host is attached to, confirm the anycast prefix is present in the RIB and that the FIB contains an ECMP entry (show ip route <anycast-prefix> and show fib <anycast-prefix>).
  4. Path verification – Trace from host to leaf (traceroute with appropriate source IP) to ensure correct encapsulation (VXLAN/MVPN) and forwarding toward the anycast next‑hop.
  5. Leaf health – If the ARP/ND MAC points to a suspect leaf, check leaf liveness (show processes cpu, show interface, show bgp summary).
  6. ECMP polarization – If traffic appears black‑holed for a subset of flows, examine the hash algorithm (show load-balancing) and consider adjusting the hash seed or using flow‑based ECMP.
  7. Rollback – If a recent policy change withdrew the anycast prefix, revert the route‑policy or re‑apply the anycast configuration on the affected leaves.

Per‑Domain Gateway

  1. Symptom – Host cannot reach default gateway; ARP/ND shows incomplete or points to wrong MAC.
  2. Determine domain – Identify the domain the host is believed to belong to (based on VLAN/VNI, EPG, or static mapping).
  3. Gateway presence – On leaves of that domain, verify the domain‑specific gateway IP is configured and advertised (show ip route <gw-ip>).
  4. Cross‑domain leakage – Check leaves outside the domain to ensure they are not advertising the same gateway IP (unintended anycast).
  5. Host ARP/ND – If the host’s ARP entry is stale, force a refresh (arp -s <gw-ip> <mac> on Linux, or clear IPv6 neighbors).
  6. Redundancy protocol state – If VRRP/HSRP is used, check the state (show vrrp, show standby) to see whether the active gateway has moved.
  7. Mobility protocol – If LISP or similar is employed, verify mapping database entries (show lisp site, show lisp ipv4 map-cache).
  8. Rollback – Correct any mis‑configured ESI or route‑policy that caused the gateway to be advertised from the wrong leaf, then re‑apply the intended domain‑scoped configuration.

Troubleshooting Complexity

FactorUniversal Leaf‑Local AnycastPer‑Domain Gateway
Number of gateway addresses to checkOneMultiple (one per domain)
Likelihood of ARP/ND stale entryLow (IP unchanged)Higher if host moves across domains without mobility protocol
Failure isolationTraffic loss usually due to leaf‑specific ECMP hash or leaf failureTraffic loss may indicate domain‑wide gateway loss or mis‑placement
ToolingStandard route/show commands; ECMP hash inspectionSame plus domain‑membership verification, redundancy‑protocol state
Root‑cause steps1) ARP/ND MAC → 2) Leaf RIB/FIB → 3) ECMP health1) Determine domain → 2) Verify gateway presence in domain → 3) Check for leakage → 4) Host ARP refresh
Automation friendlinessSimple predicate: “anycast prefix present in all leaves”Complex predicate: “prefix Pᵢ present exactly in leaves of domain Dᵢ and nowhere else”

Configuration Examples

Universal Leaf‑Local Anycast (Arista EOS – VXLAN/EVPN)

! Configure the anycast gateway IP on each leaf under the VRF
interface Vlan10
   vrf tenantA
   ip address 10.0.0.254/24 anycast-gateway   ! anycast‑gateway installs same IP on all VTEPs
   ip virtual-router address 10.0.0.254       ! optional VRRP‑like anycast behavior
!
! Advertise the anycast prefix via EVPN type‑2 (MAC‑IP) routes
router bgp 65000
   address-family l2vpn evpn
      advertise-all-vni
!
! Ensure the anycast IP is advertised as a host route (optional)
ip routing
ip route 10.0.0.254/32 Null0   ! discard locally; reachability via EVPN

Universal Leaf‑Local Anycast (Cisco NX‑OS – MPLS/VPN)

! Configure the anycast gateway IP under the VRF interface
interface Vlan10
   vrf member tenantA
   ip address 10.0.0.254/24 secondary   ! secondary enables anycast‑like behavior
   ipv6 address 2001:db8:0:10::ff/64 secondary
!
! Advertise the anycast prefix via BGP‑EVPN
router bgp 65000
   address-family l2vpn evpn
      advertise l2vpn evpn
!
! Optional static host route to trigger EVPN advertisement
ip route 10.0.0.254/32 Null0
ipv6 route 2001:db8:0:10::ff/128 Null0

Per‑Domain Gateway (Arista EOS – rack‑scoped)

! Example: rack‑1 gateway
interface Vlan10
   vrf tenantA
   ip address 10.0.1.254/24   ! unique to rack‑1
   ip virtual-router address 10.0.1.254   ! VRRP/HSRP if desired
!
! Advertise only from leaves in rack‑1 (using ESI or route‑map)
router bgp 65000
   address-family l2vpn evpn
      neighbor 10.0.0.1 route-map RACK1_OUT out
!
route-map RACK1_OUT permit 10
   match ip address prefix-list RACK1_PREFIX
   set evpn esi 00:11:22:33:44:55:

Share this post on:

Previous Post
Following a TAP frame into the host bridge
Next Post
Stabilizing noisy links without hiding real recovery time