Introduction to Chaos Engineering

Chaos Engineering is the disciplined practice of experimenting on a system to build confidence in its ability to withstand turbulent conditions in production. Instead of waiting for an unexpected failure, we deliberately inject faults—such as restarting network devices, saturating links, or disabling control‑plane services—to observe behavior, verify assumptions, and uncover hidden dependencies before they cause an outage.

The goal is not to break things for the sake of breakage but to measure resilience against a predefined steady‑state hypothesis. By defining what “normal” looks like (e.g., error rate < 0.5 %, 95th‑percentile latency < 20 ms, zero packet loss on critical paths) and then measuring deviations during the experiment, we obtain objective evidence of whether the system meets its reliability targets.

Benefits of Conducting Chaos Experiments

• Early detection of latent failure modes – Startup‑order bugs, mis‑ordered routing protocol convergence, or missing health‑check scripts surface when a subset of devices is cycled.
• Validation of automation safety – Confirms that configuration push, rollback, and verification pipelines behave correctly when the underlying state changes.
• Quantifiable blast‑radius control – Experiments are scoped to a known slice, allowing operators to predict impact and prepare mitigation.
• Improved incident response – Teams practice observing telemetry, executing runbooks, and making go/no‑go decisions under controlled stress.
• Continuous improvement feedback loop – Results feed back into design, documentation, and tooling, raising the overall maturity of network operations practice.

Designing the Chaos Experiment

Identifying Critical Network Components

Map the services that constitute the steady‑state hypothesis. For a typical IP/MPLS core, critical components include:

Select components whose restart is likely to expose startup‑order dependencies (e.g., routing protocol timers, synchronization of label distribution, or control‑plane vs. data‑plane bring‑up sequences).

Selecting a Meaningful Slice of the Network

A “meaningful slice” must be large enough to stress interactions but small enough to bound blast radius. Use these criteria:

1. Geographic containment – Choose a single Point of Presence (PoP) or a single rack within a data center.
2. Logical grouping – Select devices that share a common control‑plane protocol instance (e.g., all PE routers in a specific BGP confederation).
3. Redundancy level – Ensure at least one N‑1 redundant path remains for each service during the experiment.
4. Observability – Verify that telemetry (SNMP, streaming telemetry, syslog, flow) is available for every device in the slice.

Example slice: Four PE routers (PE1‑PE4) in PoP‑A, each dual‑homed to two P routers (P1, P2) that remain untouched. The slice represents ~10 % of total PE capacity but carries a representative mix of customer VRFs, VPNv4/VPNv6 routes, and L2VPN services.

Defining the Experiment’s Objective and Scope

• Steady‑state hypothesis: During a controlled restart of the selected PE slice, customer‑facing traffic will experience no more than a 0.5 % increase in error rate and latency will stay below the 95th‑percentile baseline of 20 ms.
• Objective: Validate that routing protocol reconvergence, LDP label distribution, and VRF bring‑up occur within expected timers and that no hidden dependencies cause prolonged black‑holes.
• Scope:
  • Devices: PE1‑PE4 (four devices).
  • Actions: Graceful reload (reload on Cisco IOS‑XR, request system reboot on Juniper Junos).
  • Duration: Maximum 15 minutes for the entire slice (including verification).
  • Blast radius: Limited to the four PEs; core P routers remain up, preserving transit for non‑slice traffic.
  • Pre‑checks: All devices report “in‑service” via SNMP ifOperStatus = up, BGP state = Established, LDP session = Operational, and no active alarms.
  • Commit boundary: The moment the reload command is issued on the first device; after this point the experiment proceeds unless a stop condition fires.
  • Verification gate: After each device reports sysUpTime reset and routing protocols re‑established, run active probes (e.g., ping/traceroute from a traffic generator) to confirm data‑plane forwarding.
  • Rollback trigger: If any stop condition is met, halt further reloads and attempt to bring any halted devices back up via reload cancel (if supported) or power‑cycle; if the device fails to respond, raise an operator intervention ticket.
  • Operator intervention point: Automatic rollback fails to restore a device within 5 minutes, or the verification gate shows sustained degradation beyond thresholds.

Implementing the Chaos Experiment

Tools and Technologies for Chaos Engineering

All tools are assumed to be already deployed in the operations pipeline; the experiment script merely calls their APIs/CLIs.

Writing the Experiment Code

Below is a self‑contained Python script that orchestrates the restart of the PE slice, evaluates stop conditions via Prometheus, and enforces the execution boundaries described earlier. The script is deliberately explicit about each phase so that an operator can audit the transaction scope.

#!/usr/bin/env python3
"""
Chaos experiment: Graceful reload of a PE slice (PE1-PE4) in PoP-A.
Implements:
  - Pre‑checks (steady‑state validation)
  - Transaction scope (slice of four devices)
  - Commit boundary (first reload command)
  - Verification gate (post‑reload health checks)
  - Rollback trigger (stop condition breach)
  - Blast radius (limited to PE devices)
  - Operator intervention point (manual ticket if auto‑recovery fails)
"""

import time
import sys
import json
import logging
from typing import List, Dict
import requests   # for Prometheus queries
from napalm import get_network_driver

# -------------------------- Configuration --------------------------
PE_DEVICES = [
    {"hostname": "pe1-popa.example.com", "username": "admin", "password": "*****", "device_type": "iosxr"},
    {"hostname": "pe2-popa.example.com", "username": "admin", "password": "*****", "device_type": "iosxr"},
    {"hostname": "pe3-popa.example.com", "username": "admin", "password": "*****", "device_type": "iosxr"},
    {"hostname": "pe4-popa.example.com", "username": "admin", "password": "*****", "device_type": "iosxr"},
]

PROMETHEUS_URL = "http://prometheus.example.com/api/v1/query"
# Steady‑state thresholds (derived from baseline)
ERROR_RATE_THRESHOLD = 0.005   # 0.5 %
LATENCY_MS_THRESHOLD = 20.0    # 95th‑pct latency in ms
PACKET_LOSS_THRESHOLD = 0.001  # 0.1 %
MAX_EXPERIMENT_SECONDS = 15 * 60   # 15 minutes
VERIFICATION_TIMEOUT = 180   # seconds to wait for each device to recover

logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(message)s")
log = logging.getLogger(__name__)

# -------------------------- Helper Functions --------------------------
def get_prometheus_query(query: str) -> float:
    """Query Prometheus and return the first scalar result."""
    resp = requests.get(PROMETHEUS_URL, params={"query": query}, timeout=10)
    resp.raise_for_status()
    data = resp.json()
    if data["status"] != "success":
        raise RuntimeError(f"Prometheus query failed: {data}")
    result = data["data"]["result"]
    if not result:
        return 0.0
    return float(result[0]["value"][1])

def steady_state_ok() -> bool:
    """Evaluate the steady‑state hypothesis via Prometheus."""
    err_rate = get_prometheus_query(
        'sum(rate(http_requests_total{job="edge",code=~"5.."}[1m])) '
        '/ sum(rate(http_requests_total{job="edge"}[1m]))'
    )
    latency = get_prometheus_query(
        'histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="edge"}[1m])) by (le))'
    )
    loss = get_prometheus_query(
        'avg by (instance) (loss_probe_ratio{job="icmp_probe"})'
    )
    log.info(f"Steady‑state metrics – error_rate={err_rate:.4f}, latency={latency:.2f}s, loss={loss:.4f}")
    return (err_rate <= ERROR_RATE_THRESHOLD and
            latency <= LATENCY_MS_THRESHOLD and
            loss <= PACKET_LOSS_THRESHOLD)

def device_precheck(dev: Dict) -> bool:
    """Verify that a device is in‑service before we touch it."""
    driver = get_network_driver(dev["device_type"])
    with driver(**dev) as device:
        device.open()
        facts = device.get_facts()
        interfaces = device.get_interfaces()
        # Simple check: all core-facing interfaces up
        core_ifs = [name for name, iface in interfaces.items()
                    if iface["is_enabled"] and iface["is_up"] and "core" in name]
        if not core_ifs:
            log.warning(f"{dev['hostname']}: No core interfaces reported up")
            return False
        log.info(f"{dev['hostname']}: Pre‑check passed ({len(core_ifs)} core interfaces up)")
        return True

def reload_device(dev: Dict) -> None:
    """Issue a graceful reload and wait for the device to come back."""
    driver = get_network_driver(dev["device_type"])
    with driver(**dev) as device:
        device.open()
        log.info(f"{dev['hostname']}: Issuing reload command")
        device.cli_command("reload")   # vendor‑specific; adjust as needed
        # After issuing reload, the connection will drop; we break out
        device.close()

def wait_for_device(dev: Dict, timeout: int = VERIFICATION_TIMEOUT) -> bool:
    """Poll until the device responds again and routing protocols are up."""
    start = time.time()
    while time.time() - start < timeout:
        try:
            driver = get_network_driver(dev["device_type"])
            with driver(**dev) as device:
                device.open()
                # Check BGP state (example for IOS‑XR)
                bgp = device.get_bgp_neighbors()
                up_peers = sum(1 for nbr in bgp.values() if nbr["is_up"])
                if up_peers > 0:
                    log.info(f"{dev['hostname']}: BGP peers up ({up_peers})")
                    # Additional checks: LDP, VRF, etc. can be added here
                    return True
        except Exception as e:
            log.debug(f"{dev['hostname']}: Not reachable yet – {e}")
        time.sleep(5)
    log.error(f"{dev['hostname']}: Did not recover within {timeout}s")
    return False

def post_reload_verification() -> bool:
    """Run active probes to verify data‑plane forwarding for the slice."""
    # Example: use a traffic generator like `trex` or `ping` from a monitor host.
    # Here we query a synthetic probe metric from Prometheus.
    loss = get_prometheus_query(
        'avg by (instance) (loss_probe_ratio{job="icmp_probe",instance=~"pe.*"} )'
    )
    latency = get_prometheus_query(
        'histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="edge",instance=~"pe.*"}[1m])) by (le))'
    )
    err = get_prometheus_query(
        'sum(rate(http_requests_total{job="edge",code=~"5..",instance=~"pe.*"}[1m])) '
        '/ sum(rate(http_requests_total{job="edge",instance=~"pe.*"}[1m]))'
    )
    log.info(f"Post‑reload verification – error_rate={err:.4f}, latency={latency:.2f}s, loss={loss:.4f}")
    return (err <= ERROR_RATE_THRESHOLD and
            latency <= LATENCY_MS_THRESHOLD and
            loss <= PACKET_LOSS_THRESHOLD)

# -------------------------- Main Experiment Flow --------------------------
def main() -> None:
    start_time = time.time()
    if not steady_state_ok():
        log.error("Steady‑state checks failed. Aborting experiment.")
        sys.exit(1)

    for idx, dev in enumerate(PE_DEVICES, start=1):
        log.info(f"=== Processing device {idx}/{len(PE_DEVICES)}: {dev['hostname']} ===")
        if not device_precheck(dev):
            log.error(f"Pre‑check failed for {dev['hostname']}. Skipping reload.")
            continue

        reload_device(dev)
        if not wait_for_device(dev):
            log.error(f"Device {dev['hostname']} did not recover. Initiating rollback.")
            # Rollback logic would go here (e.g., power‑cycle, reload cancel)
            break

        # After each reload, verify the slice still meets steady‑state
        if not steady_state_ok():
            log.error("Steady‑state violated after reload. Stopping further actions.")
            break

        # Optional: run explicit data‑plane verification
        if not post_reload_verification():
            log.error("Data‑plane verification failed. Stopping further actions.")
            break

        # Enforce overall experiment duration
        if time.time() - start_time > MAX_EXPERIMENT_SECONDS:
            log.warning("Maximum experiment duration reached.")
            break

    log.info("Experiment completed.")

if __name__ == "__main__":
    main()