When neighbor tables disagree with EVPN truth

Introduction to EVPN and Fabric Troubleshooting

EVPN carries host reachability information in the control plane using BGP NLRI. Type‑2 routes (MAC/IP advertisement) advertise a host’s MAC address, optionally paired with one or more IP addresses, and the Ethernet segment (ESI) or VNI where the host is attached.

Route fields

• Route Distinguisher (RD) – makes the prefix unique per VRF/EVI.
• Ethernet Tag ID – typically the VLAN/VNI identifier.
• MAC Address Length – 48 bits.
• MAC Address – the host’s MAC.
• IP Address Length – 0, 32, or 128 bits (0 if only MAC is advertised).
• IP Address – host IPv4 or IPv6 address when present.
• Label (VPN Label) – used for data‑plane encapsulation (VXLAN, MPLS).
• ESI – Ethernet Segment Identifier for multihoming.

These advertisements are propagated from the leaf (ToR) that learned the host via local ARP/ND or static configuration to all peers in the EVPN domain. The receiving leaf installs the MAC/IP into its local forwarding table (MAC‑VLAN table) and, if an IP is present, into its ARP/ND cache for proxy‑ARP/ND purposes.

Host‑Facing Neighbor State

Host‑facing neighbor state refers to the layer‑2/3 bindings a leaf maintains for directly attached endpoints. On most NOS this is visible as:

• ARP table (show ip arp, show ipv6 neighbors) – maps IP → MAC for IPv4/IPv6 hosts that have exchanged traffic.
• MAC address table (show mac address-table, show bridge mac-table) – maps MAC → egress port/VLAN, learned from frames seen on the interface.
• Static neighbor entries – manually configured bindings that override dynamic learning.

The neighbor state is rendered by the data plane: it reflects what the NIC has actually seen on the wire. It can diverge from the intended state (what configuration or protocols dictate) and the applied state (what the control plane has installed into the forwarding ASIC). Operators compare host‑facing neighbor state with EVPN MAC/IP advertisements to detect three conditions:

1. Fabric wrong – the advertisement is incorrect (e.g., wrong MAC/IP, missing route).
2. Stale – the advertisement is correct but outdated relative to the host’s current state (e.g., host moved, IP changed, but old advertisement still present).
3. Ahead of the host – the advertisement reflects a newer state that the host has not yet realized (e.g., ARP reply delayed, or the leaf has learned a MAC via another host’s traffic before the host itself sent a frame).

Comparing Host‑Facing Neighbor State and EVPN MAC/IP Advertisements

CLI Commands for Retrieving Neighbor State

Example (Cisco IOS):

# IPv4 ARP
show ip arp | include 10.0.1.10
# Output: Protocol  Address          Age (min)  Hardware Addr   Type   Interface
# Internet  10.0.1.10               5   aabb.cc00.0100  ARPA   GigabitEthernet1/0/1

# IPv6 ND
show ipv6 neighbors | include 2001:db8::10
# Output: IPv6 Address                              Age Link-layer Addr State Interface
# 2001:db8::10                                      0  aabb.cc00.0100  REACH GigabitEthernet1/0/1

# MAC table
show mac address-table | include aabb.cc00.0100
# Output:          Destination Address      Address Type    VLAN  Destination Port
#                aabb.cc00.0100           Dynamic         10    GigabitEthernet1/0/1

CLI Commands for Retrieving EVPN MAC/IP Advertisements

Example (Arista EOS):

# Show all MAC/IP advertisements for EVI 5000
show evpn mac ip evi 5000
# Output:
# EVI   VNI   MAC Address        IP Address        Label   ESI               Status
# 5000  5000  aabb.cc00.0100     10.0.1.10         500100  00:00:00:00:00:01 Active
# 5000  5000  aabb.cc00.0100     2001:db8::10      500100  00:00:00:00:00:01 Active

Example Code for Parsing and Comparing Neighbor State and EVPN MAC/IP Advertisements

#!/usr/bin/env python3
"""
compare_evpn_neighbor.py
Fetch host‑facing neighbor state (ARP/ND + MAC table) and EVPN MAC/IP ads,
then compare to detect fabric‑wrong, stale, or ahead conditions.
"""

import sys
import json
from napalm import get_network_driver
from collections import namedtuple

# Normalized record
Record = namedtuple("Record", ["mac", "ip", "vni"])

def normalize_mac(mac: str) -> str:
    """Convert to lowercase, colon‑separated, no leading zeros per octet."""
    return ":".join(f"{int(x, 16):02x}" for x in mac.lower().split(":"))

def get_neighbor_state(device):
    driver = get_network_driver(device["os"])
    with driver(**device["conn"]) as conn:
        conn.open()
        arp = conn.get_arp_table()          # List of dicts: interface, mac, ip, age
        nd  = conn.get_ipv6_neighbors()     # Similar structure for IPv6
        mac_table = conn.get_mac_address_table()  # List of dicts: mac, interface, vlan, static/move
        conn.close()
    # Build set of (mac, ip, vni) from ARP/ND + MAC table
    records = set()
    intf_to_vlan = {}
    for entry in mac_table:
        if entry["active"]:
            intf_to_vlan.setdefault(entry["interface"], entry["vlan"])
    # Process ARP
    for a in arp:
        mac = normalize_mac(a["mac"])
        ip = a["ip"]
        vni = intf_to_vlan.get(a["interface"], 0)
        records.add(Record(mac=mac, ip=ip, vni=vni))
    # Process IPv6 ND (store only when IP present)
    for n in nd:
        mac = normalize_mac(n["mac"])
        ip = n.get("ip")
        vni = intf_to_vlan.get(n["interface"], 0)
        if ip:
            records.add(Record(mac=mac, ip=ip, vni=vni))
    return records

def get_evpn_mac_ip(device):
    # In practice replace with live gNMI/BGP query; here we read a JSON export.
    with open(device["evpn_json"], "r") as f:
        data = json.load(f)
    records = set()
    for entry in data.get("evpn_mac_ip", []):
        mac = normalize_mac(entry["mac"])
        ip = entry.get("ip")   # may be None for MAC‑only ads
        vni = entry["vni"]
        records.add(Record(mac=mac, ip=ip, vni=vni))
    return records

def compare_sets(nbr_set, evpn_set):
    match = nbr_set & evpn_set
    fabric_wrong = evpn_set - nbr_set   # EVPN says something we do not see locally
    stale = nbr_set - evpn_set          # We see locally but EVPN does not advertise
    # Ahead detection: same MAC/VNI but different IP (or missing IP) in EVPN vs neighbor
    ahead = set()
    for e in evpn_set:
        for n in nbr_set:
            if n.mac == e.mac and n.vni == e.vni:
                if n.ip != e.ip:   # includes case where one side is None
                    ahead.add((n, e))
                break
    return match, fabric_wrong, stale, ahead

def main():
    if len(sys.argv) != 2:
        print(f"Usage: {sys.argv[0]} <device-json>")
        sys.exit(1)
    with open(sys.argv[1]) as f:
        device = json.load(f)
    nbr = get_neighbor_state(device)
    evpn = get_evpn_mac_ip(device)
    match, fw, stale, ahead = compare_sets(nbr, evpn)
    print(f"Neighbor state entries: {len(nbr)}")
    print(f"EVPN MAC/IP entries   : {len(evpn)}")
    print(f"Matches               : {len(match)}")
    print(f"Fabric‑wrong (EVPN only): {len(fw)}")
    print(f"Stale (nbr only)      : {len(stale)}")
    print(f"Ahead (IP mismatch)   : {len(ahead)}")
    if fw:
        print("\nFabric‑wrong entries:")
        for r in fw:
            print(f"  MAC={r.mac} IP={r.ip} VNI={r.vni}")
    if stale:
        print("\nStale entries (present locally, missing in EVPN):")
        for r in stale:
            print(f"  MAC={r.mac} IP={r.ip} VNI={r.vni}")
    if ahead:
        print("\nAhead entries (EVPN IP differs from neighbor):")
        for n, e in ahead:
            print(f"  Neighbor MAC={n.mac} IP={n.ip} VNI={n.vni} -> EVPN IP={e.ip}")

if __name__ == "__main__":
    main()

The script expects a device JSON file containing connection parameters (os, hostname, username, password, optional port) and a path to a pre‑collected EVPN MAC/IP JSON export (evpn_json). In production replace the file read with a live gNMI subscription or BGP rib query.

Understanding Dataplane Loss Signals

Dataplane loss manifests as packets that are dropped, discarded, or mis‑delivered before reaching the intended destination. Common signal categories:

Operators correlate these dataplane signals with the control‑plane comparisons above to pinpoint whether loss stems from fabric misconfiguration, stale state, or transient ahead‑of‑host conditions.