Underlay Is Back Overlay Is Still Lying

Introduction to Containment Failure

A containment failure occurs when the underlay network regains reachability after a disruption, but the overlay control plane—EVPN route advertisements, gateway state, or tunnel endpoint bindings—remains stale. Traffic is forwarded into a black hole because the overlay believes the next‑hop is unreachable or mis‑mapped, even though the underlying IP fabric can deliver packets. The failure is “contained” because the underlay heals while the overlay does not recover, causing silent data‑plane loss that is not reflected in typical link‑level alarms.

Causes of Containment Failure

Underlay Reachability vs. Overlay Discrepancies

Underlay Reachability Returns First

When a link or node in the IP underlay fails, IGPs (OSPF/IS‑IS) or BGP peers detect the loss via hello loss or BGP hold‑timer expiration. Upon restoration, the IGP recomputes shortest‑path trees and BGP withdraws/re‑advertises affected prefixes within its configured timers (OSPF LSA retransmission, BGP MinRouteAdvertisementInterval). The FIB is updated almost immediately, restoring IP‑level reachability between loopbacks used for VTEP underlay transport.

Overlay Gateway State Discrepancies

Overlay gateways maintain a local binding table that maps:

• MAC → VNI (learned via local ARP/ND or received EVPN type‑2 routes)
• VNI → VRF (local VNI‑to‑VRF mapping)
• VTEP IP → VNI (remote VTEP discovery via type‑3 routes)

If the underlay flaps, the gateway may:

• Flush ARP/ND entries only after the local ARP timeout (default often 4 × ARP‑retry seconds, configurable).
• Retain stale EVPN type‑2 advertisements until the local route‑refresh timer expires or an explicit withdrawal is received.
• Continue to advertise a withdrawn MAC/IP as “active” if the local proxy‑ARP/ND cache is not cleared, causing remote VTEPs to encapsulate packets to a MAC that is no longer reachable locally.

EVPN State Inconsistencies

EVPN relies on BGP to carry MAC/IP (type‑2), multicast (type‑3), and Ethernet‑segment (type‑4) routes. Inconsistencies arise when:

• A VTEP goes down; its peers withdraw type‑3 routes after the BGP hold‑timer expires, but the remote VTEP’s type‑2 routes remain in the peers’ BGP Adj‑RIB‑In until a route‑refresh or explicit withdraw is processed.
• Route‑reflector clusters retain stale routes due to missing route‑refresh capability or because the reflector client did not send a proper withdraw (e.g., due to a process crash).
• ESI‑based split‑horizon or DF election timers delay the re‑advertisement of a recovered VTEP’s Ethernet‑segment route, causing traffic to be black‑holed until the DF is re‑elected.

Tunnel Endpoint Staleness

VXLAN/GRE/VXLAN‑GPE tunnel endpoints are typically bound to a loopback address used as the underlay source/destination. Staleness can appear as:

• The tunnel interface remains administratively up but the underlying UDP destination port is unreachable because the remote loopback was withdrawn from the underlay RIB and not yet flushed from the tunnel’s destination cache.
• Missing or stale ARP/ND entries for the underlay next‑hop (the underlay next‑hop of the VTEP loopback) cause encapsulation failures even though the underlay route exists.
• Incorrect VTEP‑to‑VNI mapping in the tunnel’s decap table leading to mis‑directed inner frames.

Troubleshooting Containment Failure

Identifying Stale Overlay Gateways

1. Verify that the gateway’s local MAC‑to‑VNI table matches the EVPN type‑2 routes received from peers.
2. Check for ARP/ND entries that point to a MAC that is no longer locally attached (e.g., show arp | no-resolve on Juniper, show ip arp on Cisco).
3. Look for stale local proxy‑ARP/ND entries that persist beyond the configured ARP timeout.

Detecting EVPN State Inconsistencies

1. Compare the EVPN routing table (type‑2/3) on the VTEP with the routes advertised by its peers (show bgp evpn neighbors received-routes).
2. Verify that the Ethernet‑segment (type‑4) route for the local ESI is present and marked as “best” in the BGP table.
3. Check for route‑refresh or stale‑route flags (show bgp evpn | include stale) that indicate routes have not been refreshed after a neighbor reset.

Verifying Tunnel Endpoint Reachability

1. Confirm that the underlay route to the remote VTEP loopback exists (show route <remote-loopback>/32).
2. Validate that the tunnel interface’s destination is reachable via ping or BFD (ping <remote-loopback> source <local-loopback>).
3. Inspect the tunnel’s encapsulation cache for stale next‑hop MAC addresses (show interfaces tunnel <id> extensive on Juniper, show interface tunnel <id> on Cisco).

Example CLI Commands for Troubleshooting

Checking Underlay Reachability

# Juniper MX/QFX
show route 10.0.0.1/32 exact
show bgp summary | match 10.0.0.1

# Cisco Nexus/NX-OS
show ip route 10.0.0.1/32
show bgp ipv4 unicast summary | include 10.0.0.1

# Arista EOS
show ip route 10.0.0.1/32
show bgp summary | include 10.0.0.1

Verifying Overlay Gateway State

# Juniper (EVPN)
show evpn database mac-ip
show evpn instance default detailed | match "Local MAC"

# Cisco NX-OS
show evpn evi vni 10100 mac ip
show evpn evi vni 10100 arp-nd

# Arista
show evpn mac ip
show evpn vni 10100 arp

Displaying EVPN State

# Juniper
show bgp evpn neighbors 10.0.0.2 received-routes
show bgp evpn | match stale

# Cisco NX-OS
show bgp l2vpn evpn neighbors 10.0.0.2 received-routes
show bgp l2vpn evpn | include stale

# Arista
show bgp l2vpn evpn neighbors 10.0.0.2 received-routes
show bgp l2vpn evpn | include stale

Troubleshooting Tunnel Endpoints

# Juniper VXLAN
show interfaces vtep.0 extensive | match "Destination|State"
monitor traffic interface vtep.0 matching "udp and port 4789"

# Cisco NX-OS VXLAN
show interface nve1 peers
show interface nve1 counters

# Arista VXLAN
show interfaces vxlan1
show interfaces vxlan1 counters

Code Examples for Containment Failure Resolution

Configuring Overlay Gateways for Dynamic Updates

Juniper MX (EVPN‑VXLAN)

set routing-instances EVPN-VXLAN instance-type virtual-switch
set routing-instances EVPN-VXLAN interface lo0.0
set routing-instances EVPN-VXLAN route-distinguisher 65000:101
set routing-instances EVPN-VXLAN vrf-target target:65000:101
set routing-instances EVPN-VXLAN protocols evpn encapsulation vxlan
set routing-instances EVPN-VXLAN protocols evpn extended-vni-list 10100
set routing-instances EVPN-VXLAN vlans VLAN101 vlan-id 101
set routing-instances EVPN-VXLAN vlans VLAN101 l3-interface irb.101
set routing-instances EVPN-VXLAN vlans VLAN101 vxlan vni 10100
set protocols bgp group EVPN type internal
set protocols bgp group EVPN local-address 10.0.0.1
set protocols bgp group EVPN family evpn signaling
set protocols bgp group EVPN neighbor 10.0.0.2

The family evpn signaling statement ensures that EVPN routes are exchanged; any change in underlay reachability triggers a BGP refresh, which updates the VXLAN dataplane.

Implementing EVPN State Synchronization

Python script using Junos PyEZ to force a route‑refresh on EVPN peers when underlay flaps

#!/usr/bin/env python3
from jnpr.junos import Device
from jnpr.junos.utils.start_shell import StartShell
import sys, time

def refresh_evpn_peers(dev_ip, underlay_if):
    dev = Device(host=dev_ip, user='admin', passwd='*****')
    dev.open()
    # Detect underlay loss/gain via interface state
    with StartShell(dev) as ss:
        out = ss.run(f"show interfaces {underlay_if} terse | match {underlay_if}")
        if "up" in out:
            print(f"{dev_ip}: Underlay {underlay_if} is up – sending EVPN route‑refresh")
            dev.rpc.bgp_route_refresh(neighbor='10.0.0.2', family='evpn')
        else:
            print(f"{dev_ip}: Underlay {underlay_if} down – no action")
    dev.close()

if __name__ == '__main__':
    refresh_evpn_peers(sys.argv[1], sys.argv[2])

The script queries the underlay interface state; if it transitions to up, it issues a BGP route‑refresh for the EVPN family, causing peers to resend MAC/IP routes and eliminating stale entries.

Ensuring Tunnel Endpoint Consistency

Bash snippet to reset VXLAN tunnel destination cache when underlay route changes

#!/bin/bash
LOCAL_LO="10.0.0.1"
REMOTE_LO="10.0.0.2"
VTEP_IF="vtep.0"

while true; do
    # Check underlay route to remote loopback
    if ip route get $REMOTE_LO | grep -q "dev $LOCAL_LO"; then
        # Route exists – ensure tunnel dst is programmed
        if ! ip -d link show $VTEP_IF | grep -q "dst $REMOTE_LO"; then
            echo "$(date): Refreshing VTEP dst to $REMOTE_LO"
            ip link set dev $VTEP_IF type vxlan dstport 4789 external learning proxy \
                dstaddr $REMOTE_LO srcaddr $LOCAL_LO
        fi
    else
        echo "$(date): Underlay to $REMOTE_LO missing – bringing VTEP down"
        ip link set dev $VTEP_IF down
    fi
    sleep 5
done

This loop constantly validates that the underlay route to the remote VTEP loopback exists; if missing, it brings the VTEP interface down to prevent black‑holing, and restores it when the route returns.

Scaling Limitations and Considerations

Scalability of Overlay Gateways

• State explosion – Each VTEP maintains a MAC‑IP table proportional to the number of hosted end‑systems. In large multi‑tenant clouds, the table can exceed hardware TCAM limits, causing punt to CPU and delayed updates.
• Control‑plane load – Frequent MAC moves generate a high volume of EVPN type‑2 updates; route‑reflectors must replicate these to all clients, increasing BGP update churn.

EVPN State Synchronization Limitations

• Refresh timer dependence – Many implementations rely on configurable route‑refresh or stale‑route timers (e.g., bgp stale-routes-time). If set too high, recovery is delayed; if set too low, it creates unnecessary BGP traffic.
• Lack of explicit withdraw – Some implementations do not send an explicit withdraw when a local MAC/IP becomes unreachable, relying instead on stale‑route timers, which prolongs black‑hole periods.

End of document.