Introduction to CI Suite

Overview of Lab Environment

The lab is built with Containerlab 0.40+ using Docker 24.x as the runtime. Each node runs FRR 9.0 (or later) as the routing stack. The topology consists of two spine switches, four leaf switches, and two upstream routers that provide WAN connectivity. All nodes are Linux containers with privileged networking (--cap-add=NET_ADMIN) to allow BGP, VXLAN, and Linux bridge manipulation. The lab is instantiated via a single containerlab deploy -t topology.clab.yml command and torn down with containerlab destroy.

CI Suite Components and Tools

• Orchestrator: GitHub Actions (or GitLab CI) triggers on push to main and on pull request.
• Test runner: pytest 8.x with the pytest-dependency plugin to enforce ordered execution.
• Network automation: napalm 4.x for configuration push and ncclient for NETCONF where needed.
• Validation: FRR CLI output parsed via textfsm or ttp; optional Batfish snapshots for configuration validation.
• Evidence store: Artifacts uploaded as workflow artifacts (route tables, BGP summaries, packet captures via tcpdump).
• Failure injection: docker exec <node> ip link set dev <ethX> down or docker kill -s SIGSTOP <node> to simulate partial loss; tc qdisc add dev <ethX> root netem loss 10% for packet loss.

Designing the Lab Environment

Network Topology and Architecture

        +-----------+      +-----------+
        |  spine1   |      |  spine2   |
        +---+---+---+      +---+---+---+
            |   |              |   |
   +--------+   +--------+   +   +   +--------+
   | leaf1    leaf2    leaf3    leaf4   |
   +---+---+---+---+---+---+---+---+---+
       |   |       |   |       |   |
   +---+---+   +---+---+   +---+---+
   | uplink1|   | uplink2|   | uplink3|
   +---+---+   +---+---+   +---+---+
       |           |           |
   +---+---+   +---+---+   +---+---+
   | wan-rtr1|   | wan-rtr2|   | wan-rtr3|
   +---+---+   +---+---+   +---+---+

• Leaf switches are VTEPs; each hosts two VRFs (tenant A and tenant B) with VNI 10101 and 10102.
• Uplinks are point‑to‑point eBGP sessions between leaf and spine (AS 65001‑65004 leaf, AS 65010 spine).
• EVPN sessions are MP-BGP EVPN address‑family (AFI 25, SAFI 70) between leaf and spine, carrying MAC/IP routes.
• Tenant reachability is verified via ICMP from a host container attached to each leaf’s VRF.

Leaf, Uplink, and EVPN Session Configuration

Leaf FRR snippet (/etc/frr/frr.conf):

router bgp 65001
  bgp router-id 10.0.0.1
  neighbor spine1 interface peer-group
  neighbor spine1 remote-as 65010
  neighbor spine1 activate
  !
  address-family l2vpn evpn
    neighbor spine1 activate
    advertise-all-vni
  exit-address-family
  !
  address-family vrf tenant_a
    neighbor 10.0.10.2 remote-as 65020   ! uplink1
    neighbor 10.0.10.2 activate
    advertise ipv4 unicast
  exit-address-family

Uplink configuration mirrors the leaf with a static route to the WAN router and eBGP to the WAN AS (65100). EVPN session uses the same BGP peer group; no separate peer‑group is required.

Tenant Isolation and Reachability Requirements

• Tenant A VRF uses RD 65001:10101, RT 65001:10101 (import/export).
• Tenant B VRF uses RD 65001:10102, RT 65001:10102.
• Intra‑tenant traffic (leaf1 ↔ leaf3) must stay reachable when any single leaf, uplink, or EVPN session experiences partial loss (≤30% packet loss) but must withdraw when the underlying BGP session drops completely.
• Inter‑tenant traffic must remain blocked at all times (verified by absence of ping replies).

Implementing the CI Suite

Automated Testing Framework

The CI workflow (ci.yml) performs:

1. containerlab deploy – bring up lab.
2. wait_for_bgp – poll show bgp summary on each leaf until Established count equals expected peers.
3. baseline_snapshot – capture show ip route vrf <vrf> and show bgp l2vpn evpn as JSON.
4. Run test matrix (leaf failure, uplink failure, evpn failure) using pytest.
5. collect_evidence – gather post‑failure state, packet captures, and logs.
6. evaluate_gate – compare observed routes/traffic against pass/fail criteria.
7. containerlab destroy – cleanup.

Test Case Development for Partial Failures

All tests inherit a base class that provides:

• self.nodes: dict of node names to napalm connection objects.
• self.inject_failure(node, iface, loss=None, down=False): uses tc or ip link inside the container.
• self.verify_reachability(src_vrf, dst_host): runs ping -c 5 -W 2 from a host container and parses packet loss.

Leaf Node Failure

def test_leaf_partial_loss(self):
    # Inject 20% loss on leaf1 uplink to spine1
    self.inject_failure('leaf1', 'eth1', loss=20)
    # Wait for BGP to stay up (hold time 3s, so session stays)
    time.sleep(5)
    # Verify EVPN routes for tenant A are still present (no withdraw)
    evpn_routes = self.get_evpn_routes('leaf1', vrf='tenant_a')
    assert any(r['prefix'] == '10.0.1.0/24' for r in evpn_routes), \
        "EVPN route withdrawn despite partial loss"
    # Verify tenant A host‑to‑host ping succeeds with ≤30% loss
    loss = self.verify_reachability('tenant_a', 'host-leaf3')
    assert loss <= 30, f"Excessive loss {loss}% on tenant A"
    # Verify tenant B remains isolated (no ping)
    loss_b = self.verify_reachability('tenant_b', 'host-leaf3')
    assert loss_b == 100, "Inter‑tenant leakage detected"

Uplink Failure

def test_uplink_total_down(self):
    # Simulate complete link failure on leaf2 uplink2
    self.inject_failure('leaf2', 'eth2', down=True)
    # Wait for BGP hold timer expiry (default 180s) – we accelerate with low hold time in config
    time.sleep(20)
    # Check that IPv4 routes via uplink are removed from VRF tenant_a
    routes = self.get_ip_routes('leaf2', vrf='tenant_a')
    assert not any(r['next_hop'] == '10.0.10.2' for r in routes), \
        "Uplink route not withdrawn"
    # EVPN MAC/IP routes should stay (leaf still reachable via spine)
    evpn = self.get_evpn_routes('leaf2', vrf='tenant_a')
    assert evpn, "EVPN routes incorrectly withdrawn"
    # Tenant A reachability via spine should still work (leaf2 ↔ leaf4)
    loss = self.verify_reachability('tenant_a', 'host-leaf4')
    assert loss <= 30, "Spine path broken after uplink loss"

EVPN Session Failure

def test_evpn_session_reset(self):
    # TCP reset on BGP peer port 179 between leaf3 and spine2
    self.inject_failure('leaf3', 'eth0', down=True)  # disrupts TCP
    time.sleep(10)
    # EVPN routes must be withdrawn from leaf3 BGP table
    evpn = self.get_evpn_routes('leaf3')
    assert not evpn, "EVPN routes persisted after session drop"
    # Local VRF routes (static/connected) must remain
    vrf_routes = self.get_ip_routes('leaf3', vrf='tenant_b')
    assert vrf_routes, "VRF routes incorrectly cleared"
    # Tenant B host‑to‑host ping should fail (no VXLAN path)
    loss = self.verify_reachability('tenant_b', 'host-leaf1')
    assert loss == 100, "Tenant B reachable despite EVPN loss"

Route Withdrawal and Tenant Reachability Verification

Verification helpers:

• get_ip_routes(node, vrf): runs show ip route vrf <vrf> json via FRR’s JSON output (frr compiled with --enable-json).
• get_evpn_routes(node): runs show bgp l2vpn evpn json.
• verify_reachability(src_vrf, dst_host): executes docker exec <src-host> ping -c 5 -W 2 <dst-ip> and extracts loss from output.

Pass/fail thresholds are defined in test_config.yaml:

max_partial_loss: 30
max_total_loss: 100
bgp_hold_time: 9   # reduced