Introduction to EVPN Inconsistency Diagnosis
Overview of EVPN Technology
Ethernet VPN (EVPN) is a control‑plane technology defined in RFC 7432 that extends BGP to carry MAC‑address reachability, Ethernet segment information, and VLAN‑aware routing. It enables multi‑tenant, overlay‑based Ethernet services over an IP/MPLS core, providing features such as MAC learning suppression, aliasing, and fast convergence. In a typical deployment, each leaf (or VTEP) advertises its locally learned MAC/IP bindings via EVPN routes; the spine fabric propagates these routes so that all peers converge on a consistent view of the overlay.
Importance of Diagnosing EVPN Inconsistencies
When the EVPN control plane diverges—e.g., a leaf believes a MAC is locally attached while another leaf believes it is remote, or VNI‑to‑VLAN mappings differ—traffic can be black‑holed, duplicated, or looped. These inconsistencies are often subtle: they may affect only a subset of VNIs, appear intermittently after a topology change, or be masked by asymmetric traffic patterns. Rapid, reliable diagnosis is therefore essential to prevent service degradation, satisfy SLAs, and avoid costly manual “war‑room” efforts.
Deterministic CLI Playbooks for EVPN Diagnosis
Advantages of CLI Playbooks
- Predictability – Each step is a known CLI command; the outcome is deterministic given the same device state.
- Auditability – The exact sequence of commands can be logged, reviewed, and reproduced in a lab.
- Low dependency – Requires only an SSH/Telnet session and a basic scripting engine (e.g., Bash, Expect). No external ML inference or model‑serving infrastructure.
- Immediate feedback – Operators see raw output, which can be cross‑checked against vendor documentation.
Disadvantages of CLI Playbooks
- Manual interpretation – The operator must parse output, correlate across devices, and infer root cause.
- Scalability pain – Executing the same playbook on hundreds of leaves sequentially adds latency; parallelism introduces race conditions if state changes mid‑run.
- Brittleness to output format – Minor changes in CLI wording (e.g., a new software release) can break regex‑based parsers.
- Limited context – A playbook typically gathers a fixed set of facts; it cannot adaptively decide to collect additional evidence based on intermediate results.
Example CLI Playbook for EVPN Inconsistency Diagnosis
The following Bash playbook assumes access to a set of leaf switches via SSH (password‑less key auth). It collects core EVPN show commands, stores output in a timestamped directory, and runs a simple diff‑based check for MAC/VNI mismatches.
Code Example: CLI Playbook Script
#!/usr/bin/env bash
# evpn_diag_playbook.sh
# Usage: ./evpn_diag_playbook.sh <leaf-list-file> <output-base-dir>
# <leaf-list-file>: one hostname or IP per line
# <output-base-dir>: where per‑device directories will be created
set -euo pipefail
LEAF_LIST="${1:-leafs.txt}"
OUT_BASE="${2:-evpn_diag_$(date +%Y%m%d_%H%M%S)}"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
mkdir -p "${OUT_BASE}"
while IFS= read -r leaf; do
echo "[+] Collecting from ${leaf}"
DEV_DIR="${OUT_BASE}/${leaf}"
mkdir -p "${DEV_DIR}"
# Core EVPN show commands (vendor‑agnostic examples)
ssh -o BatchMode=yes "${leaf}" \
"show bgp l2vpn evpn route-type 2 detail; \
show bgp l2vpn evpn route-type 3 detail; \
show evpn evi detail; \
show interface status; \
show vlan brief" \
> "${DEV_DIR}/raw.txt" 2>&1 || {
echo "[-] SSH failed for ${leaf}"
continue
}
# Extract MAC‑IP bindings (type‑2) and VNI‑to‑VLAN mappings (type‑3)
awk -v outdir="${DEV_DIR}" '
/^Route Distinguisher:/ {rd=$2}
/^MAC address:/ {mac=$3}
/^IP address:/ {ip=$3}
/^ESI:/ {esi=$2}
/^Next Hop:/ {nh=$2}
/^Label:/ {label=$2}
/^Status:/ {status=$2}
/^.*/ {
if (mac && ip) {
print rd, mac, ip, esi, nh, label, status > outdir"/type2.csv"
mac=""; ip=""
}
}
' "${DEV_DIR}/raw.txt"
awk -v outdir="${DEV_DIR}" '
/^VNI:/ {vni=$2}
/^VLAN:/ {vlan=$2}
/^ESI:/ {esi=$2}
/^Next Hop:/ {nh=$2}
/^Label:/ {label=$2}
/^.*/ {
if (vni && vlan) {
print vni, vlan, esi, nh, label > outdir"/type3.csv"
vni=""; vlan=""
}
}
' "${DEV_DIR}/raw.txt"
done < "${LEAF_LIST}"
# Simple cross‑check: MAC appearing in more than one leaf with different VNIs
echo "[+] Aggregating type‑2 data..."
cat "${OUT_BASE}"/*/type2.csv 2>/dev/null | sort -u > "${OUT_BASE}/all_type2.csv"
echo "[+] Checking for MAC/VNI conflicts..."
awk '
{
key=$2; # MAC
val=$3; # IP (ignored for conflict detection)
if (seen[key] && seen[key] != val) {
print "Conflict: MAC", key, "has IP", val, "but previously seen with", seen[key];
} else {
seen[key]=val;
}
}' "${OUT_BASE}/all_type2.csv" > "${OUT_BASE}/mac_conflicts.txt" || true
echo "[+] Playbook complete. Results in ${OUT_BASE}"The script is deliberately simple: it gathers raw CLI output, extracts a few fields, and writes CSV files for later comparison. It does not attempt to remediate; it only produces evidence.
Scripted Evidence Collection for EVPN Diagnosis
Benefits of Scripted Evidence Collection
- Repeatability – A well‑versioned script (e.g., Python + Nornir) can be checked into Git, reviewed, and executed in CI pipelines.
- Rich parsing – Libraries such as
textfsm,ntc‑templates, orgeniecan transform CLI output into structured JSON/YAML, enabling programmatic correlation. - Parallel execution – Frameworks like Nornir or Ansible can query dozens of devices concurrently, reducing wall‑clock time.
- Extensibility – Adding a new show command or a new vendor is a matter of updating a template or a parser rule.
Limitations of Scripted Evidence Collection
- Dependency overhead – Requires a Python environment, library compatibility with device OS versions, and sometimes a proxy/jumphost for out‑of‑band access.
- Parser fragility – TextFSM/Genie templates must be maintained; a change in CLI wording can break parsing silently, yielding incomplete data.
- State‑change risk – Parallel collection can capture a moving target; if a leaf flaps during the window, the snapshot may be internally inconsistent.
- Operational trust – Operators must trust the parsing layer; bugs in the parser can lead to false positives/negatives that are harder to detect than a raw CLI glance.
Example Script for EVPN Evidence Collection
The following Python script uses Nornir 3.x, Netmiko, and the nornir_utils plugin for result aggregation. It retrieves EVPN type‑2 and type‑3 routes, normalizes them to a common schema, and writes a unified JSON report.
Code Example: Scripted Evidence Collection
#!/usr/bin/env python3
# evpn_evidence_collector.py
# Requires: nornir, netmiko, nornir_utils, nornir_netmiko, pyyaml
from nornir import InitNornir
from nornir_netmiko import netmiko_send_command
from nornir_utils.plugins.functions import print_result
import json
import os
from datetime import datetime
# ----------------------------------------------------------------------
# Helper: parse raw EVPN output into a list of dicts (type‑2 or type‑3)
# This example uses simple regex; in production you would plug in
# genie.parser or ntc‑templates.
# ----------------------------------------------------------------------
import re
TYPE2_RE = re.compile(
r"Route Distinguisher:\s+(?P<rd>\S+)\s+"
r"MAC address:\s+(?P<mac>\S+)\s+"
r"IP address:\s+(?P<ip>\S+)\s+"
r"ESI:\s+(?P<esi>\S+)\s+"
r"Next Hop:\s+(?P<nh>\S+)\s+"
r"Label:\s+(?P<label>\S+)\s+"
r"Status:\s+(?P<status>\S+)",
re.MULTILINE,
)
TYPE3_RE = re.compile(
r"VNI:\s+(?P<vni>\d+)\