Fault-injecting leaf-spine fabrics before production does

Introduction to Containerlab and EVPN

Overview of Containerlab

Containerlab is a declarative, Docker‑based lab orchestrator that builds multi‑node topologies from a single YAML file. Each node runs in its own container (or VM) and is connected via Linux bridges, veth pairs, or MACVLAN interfaces. The tool provides:

• Deterministic node naming (clab-<lab>-<node>).
• Automatic mgmt‑network (clab bridge) for out‑of‑band access via SSH or docker exec.
• Integration with FRR, Cisco‑like images, Linux bridges, and VXLAN via the kind: linux or kind: vr-netlab node types.
• Lifecycle commands (deploy, destroy, inspect, exec, save, restore).

A typical EVPN‑VXLAN lab uses kind: linux nodes running FRR (version ≥ 8.0) with the evpn and vxlan modules enabled. The mgmt network lets you push configurations with vtysh or ansible/nornir without touching the data plane.

EVPN Fundamentals and Inconsistencies

EVPN (RFC 7432) carries MAC/IP reachability over BGP‑NLRI type 2 (MAC‑advertisement) and type 3 (IMET). In a VXLAN‑based data plane, each leaf (VTEP) advertises:

• Local MAC/IP bindings (type 2) learned from attached hosts or ARP.
• Inclusive Multicast Ethernet Tag (type 3) to build the overlay multicast tree.

Inconsistencies arise when:

1. MAC‑move detection fails because duplicate MAC advertisements are not withdrawn.
2. Sequence numbers (in type 2) become out‑of‑sync, causing stale MACs to linger.
3. VNI‑to‑VRF mapping mismatches across peers, leading to black‑holing or leaking.
4. ESI‑based load‑balancing splits traffic unevenly when one leaf advertises a different Ethernet Segment Identifier.

These faults can stay local (affecting only the advertising leaf) or propagate (causing black‑holes, loops, or MAC‑flapping across the fabric). The experiment injects a controlled fault, observes whether the inconsistency remains confined, and verifies that the defined blast‑radius limits and stop conditions contain the impact.

Defining Steady-State Expectations

Establishing Baseline Network Behavior

Before fault injection, capture a reproducible steady state:

• Control‑plane convergence – all BGP EVPN peers are in Established state, no Active or Idle.
• Data‑plane forwarding – ping/trace between any two hosts succeeds with ≤ 1 ms jitter and zero loss.
• MAC table symmetry – each leaf’s MAC table (show mac evpn) contains exactly the MACs advertised by the remote leaves for its VNI.
• VXLAN encapsulation – tcpdump -i vxlan10 shows VXLAN packets with correct VNI and UDP dst port 4789.

Collect these metrics via a script that runs vtysh -c "show bgp evpn summary" and show mac evpn on every leaf, then stores JSON output for later comparison.

Identifying Key Performance Indicators (KPIs)

These KPIs form the basis for stop‑condition logic: if any KPI deviates beyond a tolerance, the experiment halts.

Creating Steady-State Scenarios in Containerlab

Define a base topology (clab-evpn-base.yml) with four leaf nodes (leaf1..leaf4) and two spine nodes (spine1..spine2). Each leaf connects to both spines via 10 GbE virtual links (kind: veth). All nodes run the same FRR image (frrouting/frr:8.4).

name: clab-evpn-base
topology:
  nodes:
    leaf1:
      kind: linux
      image: frrouting/frr:8.4
      cmd: /usr/lib/frr/zebra -d && /usr/lib/frr/bgp -d
      binds:
        - ./frr/leaf1/frr.conf:/etc/frr/frr.conf
        - ./frr/leaf1/daemons:/etc/frr/daemons
    leaf2:
      kind: linux
      image: frrouting/frr:8.4
      cmd: /usr/lib/frr/zebra -d && /usr/lib/frr/bgp -d
      binds:
        - ./frr/leaf2/frr.conf:/etc/frr/frr.conf
        - ./frr/leaf2/daemons:/etc/frr/daemons
    leaf3:
      kind: linux
      image: frrouting/frr:8.4
      cmd: /usr/lib/frr/zebra -d && /usr/lib/frr/bgp -d
      binds:
        - ./frr/leaf3/frr.conf:/etc/frr/frr.conf
        - ./frr/leaf3/daemons:/etc/frr/daemons
    leaf4:
      kind: linux
      image: frrouting/frr:8.4
      cmd: /usr/lib/frr/zebra -d && /usr/lib/frr/bgp -d
      binds:
        - ./frr/leaf4/frr.conf:/etc/frr/frr.conf
        - ./frr/leaf4/daemons:/etc/frr/daemons
    spine1:
      kind: linux
      image: frrouting/frr:8.4
      cmd: /usr/lib/frr/zebra -d && /usr/lib/frr/bgp -d
      binds:
        - ./frr/spine1/frr.conf:/etc/frr/frr.conf
        - ./frr/spine1/daemons:/etc/frr/daemons
    spine2:
      kind: linux
      image: frrouting/frr:8.4
      cmd: /usr/lib/frr/zebra -d && /usr/lib/frr/bgp -d
      binds:
        - ./frr/spine2/frr.conf:/etc/frr/frr.conf
        - ./frr/spine2/daemons:/etc/frr/daemons
  links:
    - endpoints: ["leaf1:eth1", "spine1:eth1"]
    - endpoints: ["leaf1:eth2", "spine2:eth1"]
    - endpoints: ["leaf2:eth1", "spine1:eth2"]
    - endpoints: ["leaf2:eth2", "spine2:eth2"]
    - endpoints: ["leaf3:eth1", "spine1:eth3"]
    - endpoints: ["leaf3:eth2", "spine2:eth3"]
    - endpoints: ["leaf4:eth1", "spine1:eth4"]
    - endpoints: ["leaf4:eth2", "spine2:eth4"]
    - endpoints: ["leaf1:eth3", "leaf2:eth3"]   # host-facing
    - endpoints: ["leaf3:eth3", "leaf4:eth3"]   # host-facing

Deploy with containerlab deploy -t clab-evpn-base.yml. After the containers are up, run a bootstrap script that:

1. Configures each leaf’s loopback (10.0.0.X/32) and VNI (10000).
2. Enables EVPN (address-family l2vpn evpn) and advertises the loopback as the BGP update source.
3. Starts a lightweight traffic generator (iperf3 in server mode on each host‑facing namespace).

Once the script reports all KPIs within target ranges, you have a verified steady state.

Setting Blast-Radius Limits

Understanding Blast Radius and Its Impact

In a partial‑failure experiment, blast radius is the set of nodes whose operational state may deviate from steady state as a direct or indirect consequence of the injected fault. Limiting blast radius ensures that a fault does not cascade to unintended parts of the fabric (e.g., causing a spine‑wide BGP flap).

Containerlab does not provide a native “blast‑radius” knob; instead, you enforce limits by:

• Scoping the fault injection to a subset of nodes via clab exec or docker exec.
• Isolating failure effects using Linux traffic control (tc) or iptables rules that drop or delay packets only on selected interfaces.
• Monitoring the rest of the fabric and triggering a stop condition if any KPI deviates outside the defined tolerance.

Configuring Blast-Radius Limits in Containerlab

Define a failure‑profile YAML that lists the nodes and interfaces allowed to be perturbed. The experiment driver reads this file and applies the fault only to those entries.

Example failure-profile.yml:

blast_radius:
  nodes:
    - leaf1
    - leaf2
  interfaces:
    leaf1:
      - eth1   # link to spine1
      - eth2   # link to spine2
    leaf2:
      - eth1   # link to spine1
      - eth2   # link to spine2

The driver will only apply tc rules, interface shutdowns, or FRR shutdown commands on those interfaces.

Example CLI Commands for Blast-Radius Configuration

Assuming you have deployed the lab and want to limit the blast radius to the two leaf‑to‑spine links of leaf1 and leaf2:

# 1. Identify the veth pair names created by Containerlab
clab inspect -t clab-evpn-base | grep -E 'leaf1:eth1|leaf1:eth2|leaf2:eth1|leaf2:eth2'

# Sample output (your names will differ):
# leaf1-eth1  <->  spine1-eth1   veth
# leaf1-eth2  <->  spine2-eth1   veth
# leaf2-eth1  <->  spine1-eth2   veth
# leaf2-eth2  <->  spine2-eth2   veth

# 2. Apply a 10 % loss, 5 ms delay on those veth pairs (tc netem)
for iface in leaf1-eth1 leaf1-eth2 leaf2-eth1 leaf2-eth2; do
  sudo tc qdisc add dev $iface root netem loss 10% delay 5ms
done

To limit the blast radius to a single node (e.g., only leaf1), simply omit the other nodes from the profile and run the same tc commands on its interfaces only.

To reset (remove the limit) after the experiment:

for iface in leaf1-eth1 leaf1-eth2 leaf2-eth1 leaf2-eth2; do
  sudo tc qdisc del dev $iface root
done

Establishing Stop Conditions

Defining Stop Conditions for Partial-Failure Experiments

A stop condition is a predicate that, when true, triggers an automatic halt of the experiment and initiates recovery. For EVPN labs we recommend the following hierarchical stop conditions:

1. Hard stop – any spine node loses BGP EVPN adjacency with all its leaf peers (show bgp evpn summary shows Idle/Active).
2. Soft stop – any leaf’s MAC‑advertisement count deviates by > 20 % from baseline or RTT exceeds 5 ms avg for three consecutive samples.
3. Warning – packet loss on any monitored link > 5 % (used for logging only).

The experiment driver evaluates these conditions every 2 seconds. If a hard stop fires, the driver immediately issues containerlab destroy (or a targeted clab exec to shut down the offending BGP peers). If only a soft stop fires, the driver attempts a graceful recovery (e.g., removes tc netem, issues clear bgp evpn *).

Implementing Stop Conditions in Containerlab

Containerlab itself does not run a monitoring loop; you orchestrate it from the host with a simple bash/python script that:

• SSHs into the mgmt network (clab bridge) to run vtysh commands.
• Parses JSON output (show bgp evpn summary json).
• Compares against thresholds.

Because the mgmt network is isolated, the script does not interfere with the data plane.

Code Examples for Automating Stop Conditions

Below is a Python 3.8+ script (evpn_monitor.py) that implements the stop‑condition logic. It assumes the lab name is clab-evpn-base and that the mgmt network is reachable via 172.20.20.0/24 (Containerlab’s default).

#!/usr/bin/env python3
import subprocess, time, json, sys, os

LAB = "