Review of Architectural Trade‑offs of Stretched L2 Domains Across Leaf‑Spine Fabrics

Engineering design‑review memo focused on blast‑radius containment when a fabric segment degrades.

Introduction

A leaf‑spine fabric is a two‑tier Clos topology where every leaf connects to every spine, delivering deterministic, non‑blocking bandwidth and predictable latency. The fabric is inherently L3‑oriented: spines forward IP packets via ECMP, while leaves provide access functions (servers, storage, uplinks).

When the fabric transports stretched L2 domains—a single VLAN or VXLAN segment spanning multiple leaves—the L2 forwarding plane extends beyond traditional rack or campus boundaries. Common motivations include:

• vMotion / live VM migration requiring L2 adjacency.
• Legacy applications that rely on broadcast‑based discovery (e.g., certain storage protocols).
• Disaster‑recovery sites needing the same subnet in two locations.

Benefits: application transparency, simplified migration, unified policy.
Cost: the L2 failure domain is stretched across the fabric, directly influencing blast radius when a segment degrades.

Architectural Trade‑offs

Design Complexity

• Control‑plane proliferation – Pure L2 trunking requires VLAN pruning and MAC‑learning across spines; overlay approaches (VXLAN/EVPN) add VTEP reachability, EVPN MAC/IP routes, and optional ARP/ND suppression.
• Policy boundaries – L2 ACLs applied at leaf ingress/egress must account for possible hair‑pinning through spines; mismatches cause black‑holing or leakage.
• Operational overhead – Documentation must track stretched VLAN/VNI, participating leaves, and any MSTP/VXLAN‑specific parameters (VNI‑to‑VLAN mapping, VTEP IPs).

Convergence and Stability

• STP/MSTP – With a VLAN trunked across spines, BPDUs traverse the entire fabric. A topology change notice (TCN) on any leaf triggers global reconvergence (all switches recompute root, port states, and may block ports), causing a fabric‑wide forwarding pause (~2× forward delay + max age ≈ 30‑50 s with default timers). Using MSTP with multiple instances or PVST+ (per‑VLAN STP) limits BPDU scope to the relevant MSTI/VLAN.
• EVPN MAC mobility – A MAC move generates a MAC‑Mobility Extended Community update. Flapping MACs cause repeated BGP withdraw/re‑advertise, leading to route‑flap damping and possible control‑plane overload on spine route‑reflectors.
• Broadcast/unknown‑unicast flooding – In VXLAN, broadcast frames are replicated by head‑end or ingress replication. A broadcast storm on one leaf is therefore sent to every VTEP that has joined the VNI, turning a local fault into a fabric‑wide bandwidth consumer.

Traffic Engineering and Load Balancing

• ECMP hash polarization – Pure L2 trunk load‑balances on src/dst MAC (or VLAN ID). Low MAC address variety (e.g., many VMs sharing an OUI) can skew the hash, overloading specific spine links.
• Overlay encapsulation overhead – VXLAN adds 50 bytes (outer UDP+IP+VXLAN). Large‑MTU traffic (iSCSI, NFS) sees payload MTU drop to ~1450 bytes, potentially triggering IP fragmentation or TCP MSS reduction, affecting throughput and retransmission timers.
• Hot‑potato routing – With EVPN, the optimal exit leaf for a destination MAC is chosen from advertised MAC‑IP routes. If the route‑reflector prefers a leaf with longer physical latency (AS‑path length, local‑pref), traffic may experience sub‑optimal pathing and increased jitter.

Blast‑Radius Analysis

Defining Blast Radius

Blast radius = set of forwarding entities (ports, VTEPs, spines) whose data‑plane behavior changes directly due to a fault in a single fabric segment (leaf, link, or VLAN/VNI). In a stretched L2 domain the radius can be:

• Local – confined to the originating leaf (e.g., port‑level error that does not generate BPDUs or MAC withdrawals).
• Regional – spreads to all leaves sharing the same VLAN/VNI but stays within the fabric (e.g., broadcast storm replicated via VXLAN).
• Global – affects the entire fabric, including spine forwarding tables and control‑plane protocols (e.g., STP TCN triggering fabric‑wide reconvergence).

Factors Influencing Containment

Local vs. Global Impact of Segment Degradation

Key insight: Blast radius expands in proportion to the amount of control‑plane state that crosses the fabric for the given L2 segment. Pure L2 trunking propagates STP BPDUs and broadcast frames indiscriminately, yielding the largest radius. Encapsulation with VXLAN/EVPN confines the radius to the VNI scope, provided the overlay control plane (BGP) remains stable and storm‑control/MAC‑learning limits are enforced.

Troubleshooting Stretched L2 Domain Issues

Identifying Segment‑Degradation Symptoms

CLI Commands for Monitoring and Debugging

Cisco Nexus 9000 (NX‑OS) – VXLAN/EVPN

# Verify VNI membership and VTEP reachability
show nve vni
show nve peers

# MAC‑learning table for a specific VNI
show mac address-table vni <vni-id>

# EVPN MAC/IP routes
show bgp l2vpn evpn route-type mac-ip

# STP status per VLAN (if VLAN trunked)
show spanning-tree vlan <vlan-id> detail

# Broadcast storm counters
show interface <if> counters storm-control

# NVE (VXLAN) encapsulation errors
show nve interface nve1 statistics

Arista EOS – VXLAN/EVPN

# VTEP and VNI status
show vxlan vni
show vxlan tunnel

# MAC address table per VNI
show mac address-table vni <vni-id>

# EVPN routes
show bgp l2vpn evpn

# STP (MSTP) detail
show spanning-tree mst detail

# Storm‑control configuration and counters
show storm-control interface Ethernet<num/num>

# Interface error counters
show interfaces Ethernet<num/num> counters

Example: Automated Fault Detection (Napalm)

The following Python script polls a leaf for MAC‑table utilization and raises an alarm if usage exceeds 80 % for a given VNI. It can be run via cron or integrated into an Ansible playbook.

#!/usr/bin/env python3
import sys
from napalm import get_network_driver
from collections import defaultdict

DRIVER = get_network_driver('iosxr')   # or 'eos', 'nxos'
DEVICE = {
    'hostname': 'leaf01.example.com',
    'username': 'admin',
    'password': '*****',
    'optional_args': {'port': 830}
}

VNI_OF_INTEREST = 10100
THRESHOLD_PCT = 80

def main():
    dev = DRIVER(**DEVICE)
    dev.open()
    mac_table = dev.get_mac_address_table()
    dev.close()

    # Count entries per VNI
    vni_counts = defaultdict(int)
    for entry in mac_table:
        # entry format varies; assume 'vlan' field maps to VNI in VXLAN case
        vni = entry.get('vlan') or entry.get('vni')
        if vni is not None:
            vni_counts[vni] += 1

    total = vni_counts.get(VNI_OF_INTEREST, 0)
    # Assume a fixed MAC‑table size; adjust as needed for your platform
    MAC_TABLE_SIZE = 32768
    usage_pct = (total / MAC_TABLE_SIZE) * 100

    if usage_pct > THRESHOLD_PCT:
        print(f"ALARM: VNI {VNI_OF_INTEREST} MAC-table usage {usage_pct:.1f}% exceeds {THRESHOLD_PCT}%")
        sys.exit(1)
    else:
        print(f"OK: VNI {VNI_OF_INTEREST} MAC-table usage {usage_pct:.1f}%")
        sys.exit(0)

if __name__ == '__main__':
    main()

End of memo.