Introduction
Stretched Layer‑2/Layer‑3 fabrics (EVPN‑VXLAN, MPLS‑based DCI) need a default‑gateway address reachable from any leaf where a workload may reside. Two common designs are:
- Universal leaf‑local anycast gateway – the same anycast IP (and MAC, if L2) is configured on every leaf.
- Per‑domain gateway placement – distinct gateway IPs are assigned to smaller failure domains (rack, pod, aggregation block); only leaves in that domain advertise the address.
Both provide first‑hop routing while preserving mobility, but they differ in control‑plane state, failure domains, and operational model. This memo compares them across mobility scope, blast radius, operational clarity, troubleshooting, and provability, then offers guidance for selecting the design that matches operator intent.
Mobility Scope
Universal Leaf‑Local Anycast
- Definition – Set of locations where a workload retains its original IP and default‑gateway reachability without re‑IP or ARP/ND change.
- Mechanism – The anycast IP appears on every leaf; moving a workload from leaf A to leaf B leaves the gateway IP unchanged in the host’s ARP/ND cache. Only the local MAC binding changes, handled locally by the leaf’s ARP/ND table.
- Control‑plane impact – The anycast prefix is advertised via BGP‑EVPN type‑2 routes with the same ESI on all leaves. Host‑based ARP/ND resolution does not need refreshing on mobility.
- Result – Mobility scope is fabric‑wide (any leaf in the same VNI/VRF). Workloads can roam across pods, racks, or sites without IP‑level disruption, assuming end‑to‑end overlay reachability.
Per‑Domain Gateway
- Definition – Mobility scope limited to the boundaries of the domain that owns the gateway address.
- Mechanism – Each domain advertises its own gateway IP via a unique ESI or a domain‑scoped anycast prefix. When a workload moves to a leaf outside its home domain, the original gateway IP is no longer locally attached; the host must resolve a new gateway address (ARP/ND, DHCP, or a mobility protocol).
- Control‑plane impact – Multiple gateway prefixes are carried in the underlay, each tied to a specific leaf set. Mobility triggers a change in the host’s default‑gateway IP unless an external mobility solution rewrites the host routing table or intercepts ARP/ND.
- Result – Mobility scope is domain‑local (typically a rack or pod). Cross‑domain movement requires host re‑IP, a mobility protocol (LISP‑mobile node, proxy‑ARP), or a gateway‑redirection service (VRRP/HSRP with active‑standby across domains).
Implications
| Aspect | Universal Leaf‑Local Anycast | Per‑Domain Gateway |
|---|
| Maximum mobility | Fabric‑wide (any leaf in VNI/VRF) | Limited to domain boundary |
| Host‑side state change on move | Only MAC binding changes; IP unchanged | Default‑gateway IP may change |
| Need for host mobility protocol | Generally none | Often required for seamless cross‑domain move |
| ARP/ND impact | Local MAC update; no ARP refresh needed | New ARP/ND resolution for new gateway IP |
| Operational assumption | Workloads retain same IP/gateway across fabric | Workloads assumed static within a domain |
Blast Radius
Universal Leaf‑Local Anycast
- Failure domain – The anycast IP is present on every leaf; loss of a single leaf does not remove the gateway address because other leaves still advertise the same prefix.
- Control‑plane effect – Withdrawal of the anycast route from a failed leaf triggers BGP‑EVPN path‑selection recomputation, but the prefix remains reachable via the remaining leaves. Convergence follows the underlying routing protocol (BGP hold time ≈90 s, faster with BGP‑add‑path/PIC).
- Data‑plane effect – Traffic to the anycast IP is load‑balanced (ECMP or flow‑based) across healthy leaves. A leaf failure redirects its share; no black‑hole occurs unless all leaves advertising the prefix fail simultaneously.
- Scope of impact – Blast radius limited to traffic hashed to the failed leaf; a mis‑configuration that withdraws the prefix from all leaves causes a fabric‑wide gateway outage.
Per‑Domain Gateway
- Failure domain – Each domain owns a distinct gateway prefix. Failure of all leaves in a domain removes that gateway address from the fabric.
- Control‑plane effect – When the last leaf in a domain withdraws its gateway prefix, hosts in that domain lose reachability to their default gateway unless a redundancy protocol (VRRP/HSRP) provides an alternate active gateway in another domain (different IP).
- Data‑plane effect – Traffic to the failed gateway is dropped until hosts learn a new gateway address (ARP/ND, DHCP, or mobility protocol). With a first‑hop redundancy protocol, blast radius limited to switchover time (sub‑second to a few seconds).
- Scope of impact – A leaf failure affects traffic only if it is the last leaf in its domain; otherwise other leaves in the same domain continue to provide the gateway. The design creates multiple independent failure domains, each with its own blast radius. A domain‑wide event (e.g., rack power loss) disables the gateway for that entire domain.
Implications
| Aspect | Universal Leaf‑Local Anycast | Per‑Domain Gateway |
|---|
| Single leaf failure | No gateway loss; traffic rebalanced | No impact unless leaf is last in its domain |
| Domain‑wide failure (rack power loss) | Gateway still reachable via other leaves | Gateway lost for that domain; hosts lose default gateway |
| Correlated failures | Requires simultaneous loss of all leaves to break gateway | Any failure that empties a domain breaks gateway for that domain |
| Recovery mechanism | Automatic via routing reconvergence; no host action | May require ARP/ND refresh, DHCP renewal, or mobility protocol |
| Blast‑radius predictability | Uniform across fabric; depends on number of leaves advertising prefix | Heterogeneous; each domain has its own radius |
Operational Clarity
Universal Leaf‑Local Anycast
- Visibility – The anycast prefix appears as a single RIB entry with multiple next‑hops (one per leaf).
show ip route <anycast-prefix> lists ECMP next‑hops, making it easy to verify that the prefix is advertised from the expected leaf set.
- Policy tracing – Because the same prefix is used everywhere, route‑policy checks need to be performed only once per prefix; there is no risk of a leaf advertising a different gateway address.
- Configuration drift detection – A simple compliance check verifies that every leaf in the fabric has the anycast IP configured under the appropriate VRF/VNI. Any deviation appears as a missing next‑hop in the ECMP set.
- Auditability – Proving “the gateway address is reachable from any leaf” reduces to showing the anycast prefix is present in the RIB/FIB of every leaf and that the forwarding table contains an ECMP entry—scriptable with minimal state.
- Complexity – Operators must understand ECMP load‑balancing and anycast behavior (flow‑based hashing, potential polarization), but only one gateway address per VNI/VRF needs tracking.
Per‑Domain Gateway
- Visibility – Each domain introduces its own gateway prefix. Operators must maintain a mapping of domain → gateway IP → set of leaves. The routing table shows multiple distinct prefixes, each with its own ECMP set (often a single next‑hop if only one leaf in the domain advertises it).
- Policy tracing – Route‑policies may need to be scoped per domain (e.g., different import/export rules for each gateway prefix), increasing the number of policy statements and chance of mismatch.
- Configuration drift detection – Drift can occur at two levels: (1) missing gateway IP on a leaf that should belong to the domain, and (2) accidental advertisement of a gateway IP from a leaf outside its intended domain (mis‑configured ESI or route‑policy). Detecting the latter requires cross‑checking the advertised prefix against leaf domain membership.
- Auditability – Proving “hosts in domain X always use gateway Gₓ” requires verifying both (a) that Gₓ is present on all leaves of domain X and absent from leaves outside X, and (b) that hosts’ ARP/ND caches point to Gₓ—more state to validate.
- Complexity – Operators must track a gateway‑to‑domain mapping, manage multiple anycast prefixes, and ensure redundancy protocols (if used) do not create conflicting gateways. The model resembles traditional multi‑gateway designs.
Implications
| Aspect | Universal Leaf‑Local Anycast | Per‑Domain Gateway |
|---|
| Distinct gateway prefixes | 1 per VNI/VRF | N (one per domain) |
| ECMP next‑hop count | Equal to number of leaves advertising prefix (often large) | Usually 1 per domain (if single leaf) or small set |
| Policy scope | Single prefix → simple | Multiple prefixes → per‑domain policies |
| Drift detection | Verify uniform presence/absence | Verify per‑domain presence and cross‑domain absence |
| Proof of reachability | Show ECMP set in every leaf | Show prefix present only in intended domain’s leaves |
| Operational overhead | Low (single address) | Higher (address book, mapping, redundancy protocols) |
Troubleshooting
Universal Leaf‑Local Anycast
- Symptom – Host cannot reach default gateway (ping fails).
- First check – Verify host’s ARP/ND entry for the gateway IP resolves to a MAC belonging to a leaf (
show arp / show ipv6 neighbors).
- Next‑hop validation – On the leaf the host is attached to, confirm the anycast prefix is present in the RIB and that the FIB contains an ECMP entry (
show ip route <anycast-prefix> and show fib <anycast-prefix>).
- Path verification – Trace from host to leaf (
traceroute with appropriate source IP) to ensure correct encapsulation (VXLAN/MVPN) and forwarding toward the anycast next‑hop.
- Leaf health – If the ARP/ND MAC points to a suspect leaf, check leaf liveness (
show processes cpu, show interface, show bgp summary).
- ECMP polarization – If traffic appears black‑holed for a subset of flows, examine the hash algorithm (
show load-balancing) and consider adjusting the hash seed or using flow‑based ECMP.
- Rollback – If a recent policy change withdrew the anycast prefix, revert the route‑policy or re‑apply the anycast configuration on the affected leaves.
Per‑Domain Gateway
- Symptom – Host cannot reach default gateway; ARP/ND shows incomplete or points to wrong MAC.
- Determine domain – Identify the domain the host is believed to belong to (based on VLAN/VNI, EPG, or static mapping).
- Gateway presence – On leaves of that domain, verify the domain‑specific gateway IP is configured and advertised (
show ip route <gw-ip>).
- Cross‑domain leakage – Check leaves outside the domain to ensure they are not advertising the same gateway IP (unintended anycast).
- Host ARP/ND – If the host’s ARP entry is stale, force a refresh (
arp -s <gw-ip> <mac> on Linux, or clear IPv6 neighbors).
- Redundancy protocol state – If VRRP/HSRP is used, check the state (
show vrrp, show standby) to see whether the active gateway has moved.
- Mobility protocol – If LISP or similar is employed, verify mapping database entries (
show lisp site, show lisp ipv4 map-cache).
- Rollback – Correct any mis‑configured ESI or route‑policy that caused the gateway to be advertised from the wrong leaf, then re‑apply the intended domain‑scoped configuration.
Troubleshooting Complexity
| Factor | Universal Leaf‑Local Anycast | Per‑Domain Gateway |
|---|
| Number of gateway addresses to check | One | Multiple (one per domain) |
| Likelihood of ARP/ND stale entry | Low (IP unchanged) | Higher if host moves across domains without mobility protocol |
| Failure isolation | Traffic loss usually due to leaf‑specific ECMP hash or leaf failure | Traffic loss may indicate domain‑wide gateway loss or mis‑placement |
| Tooling | Standard route/show commands; ECMP hash inspection | Same plus domain‑membership verification, redundancy‑protocol state |
| Root‑cause steps | 1) ARP/ND MAC → 2) Leaf RIB/FIB → 3) ECMP health | 1) Determine domain → 2) Verify gateway presence in domain → 3) Check for leakage → 4) Host ARP refresh |
| Automation friendliness | Simple predicate: “anycast prefix present in all leaves” | Complex predicate: “prefix Pᵢ present exactly in leaves of domain Dᵢ and nowhere else” |
Configuration Examples
Universal Leaf‑Local Anycast (Arista EOS – VXLAN/EVPN)
! Configure the anycast gateway IP on each leaf under the VRF
interface Vlan10
vrf tenantA
ip address 10.0.0.254/24 anycast-gateway ! anycast‑gateway installs same IP on all VTEPs
ip virtual-router address 10.0.0.254 ! optional VRRP‑like anycast behavior
!
! Advertise the anycast prefix via EVPN type‑2 (MAC‑IP) routes
router bgp 65000
address-family l2vpn evpn
advertise-all-vni
!
! Ensure the anycast IP is advertised as a host route (optional)
ip routing
ip route 10.0.0.254/32 Null0 ! discard locally; reachability via EVPN
Universal Leaf‑Local Anycast (Cisco NX‑OS – MPLS/VPN)
! Configure the anycast gateway IP under the VRF interface
interface Vlan10
vrf member tenantA
ip address 10.0.0.254/24 secondary ! secondary enables anycast‑like behavior
ipv6 address 2001:db8:0:10::ff/64 secondary
!
! Advertise the anycast prefix via BGP‑EVPN
router bgp 65000
address-family l2vpn evpn
advertise l2vpn evpn
!
! Optional static host route to trigger EVPN advertisement
ip route 10.0.0.254/32 Null0
ipv6 route 2001:db8:0:10::ff/128 Null0
Per‑Domain Gateway (Arista EOS – rack‑scoped)
! Example: rack‑1 gateway
interface Vlan10
vrf tenantA
ip address 10.0.1.254/24 ! unique to rack‑1
ip virtual-router address 10.0.1.254 ! VRRP/HSRP if desired
!
! Advertise only from leaves in rack‑1 (using ESI or route‑map)
router bgp 65000
address-family l2vpn evpn
neighbor 10.0.0.1 route-map RACK1_OUT out
!
route-map RACK1_OUT permit 10
match ip address prefix-list RACK1_PREFIX
set evpn esi 00:11:22:33:44:55: