Deliberately isolating a route reflector to calibrate budgets

Introduction to Route-Reflector Isolation Experiment

Overview of Route-Reflector Architecture

A route reflector (RR) is a BGP speaker that relaxes the full‑mesh requirement inside an autonomous system (AS) by allowing client peers to advertise routes to the RR, which then re‑advertises those routes to other clients and non‑client peers. The RR maintains a cluster‑ID (either configured explicitly or derived from its BGP router‑ID) and attaches the ORIGINATOR_ID attribute to prevent looping.

Key invariants

These rules guarantee that, in a stable topology, the set of routes known to each BGP speaker converges to the same global view (modulo local preference).

Importance of Controlled Experiments in Networking

Network behavior under stress is often inferred from production incidents, which conflates multiple failure domains. A controlled experiment isolates a single variable—here, the availability or advertisement capability of a route reflector—while keeping all other factors (link state, IGP metrics, traffic load) constant. This enables:

• Precise measurement of convergence time and update churn attributable to the RR isolation.
• Establishment of a baseline “steady‑state” hypothesis against which deviations can be judged.
• Identification of thresholds where expected degradation (e.g., temporary sub‑optimal paths) transitions into an unexpected control‑plane cascade (e.g., session flaps, memory exhaustion).

By defining stop conditions and convergence budgets a priori, the experiment yields repeatable, actionable data for capacity planning and failure‑mode analysis.

Defining Steady‑State Expectations

Establishing Baseline Network Performance

Before injecting any fault, the network must be in a known, stable condition. The baseline is captured by:

1. BGP session state – All peers in Established state (show bgp summary).
2. RIB/FIB consistency – Number of prefixes in the BGP RIB equals the number of prefixes installed in the FIB (show ip route bgp vs show ip route).
3. Update rate – Average BGP update messages per second over a 5‑minute window (show bgp process | include Updates).
4. CPU/Memory utilization – Router CPU < 30 % and BGP process memory < 60 % of allocated (platform‑specific counters).
5. Traffic loss – Zero packet loss for synthetic probes (<0.01 % loss over 10 s intervals).

These metrics constitute the steady‑state hypothesis: If no configuration or topology change occurs, the above metrics will remain within ±5 % of their observed values for at least 15 minutes.

Identifying Key Performance Indicators (KPIs)

Setting Realistic Steady‑State Expectations

Using the baseline, we define acceptable bands:

• T₍conv₎ ≤ baseline × 1.5 (allowing for modest processing delay).
• U₍churn₎ ≤ baseline × 2.0 (tolerating bursty reflections).
• ΔCPU ≤ 20 % absolute.
• ΔMem ≤ 100 MB absolute.
• POR ≥ 0.90 (some temporary sub‑optimal routing is expected while reflected routes are withdrawn).

If any KPI exceeds its band, the observation is flagged as potential anomaly and subjected to root‑cause analysis.

Designing the Isolation Experiment

Selecting Isolation Scenarios and Parameters

Three orthogonal isolation vectors are useful:

For a first‑order study we select RR Shutdown because it yields the clearest binary state (RR present vs absent) and isolates the reflection function entirely.

Parameters to fix:

• IGP metric – Keep OSPF/IS‑IS costs static (no link flaps).
• Traffic load – Use a constant‑bitrate traffic generator (e.g., IXIA, TRex) to produce 1 Gbps of mixed TCP/UDP flows across the AS.
• Measurement interval – Collect KPI samples every 5 seconds.

Configuring Route-Reflectors for Isolation

Below are CLI snippets for the three most common network operating systems. The isolation step is simply shutting down the BGP neighbor relationships or the entire BGP process.

Cisco IOS‑XR

# Baseline RR configuration
router bgp 65000
 bgp router-id 10.0.0.1
 address-family ipv4 unicast
  route-reflector-cluster-id 10.0.0.1
 !
 neighbor 10.0.0.2
  remote-as 65000
  address-family ipv4 unicast
   route-reflector-client
  !
 !
 neighbor 10.0.0.3
  remote-as 65000
  address-family ipv4 unicast
   route-reflector-client
  !

Isolation (shutdown RR):

router bgp 65000
 shutdown   # disables the entire BGP process

Alternatively, to isolate only the reflection function while keeping the RR up for monitoring:

router bgp 65000
 neighbor 10.0.0.2 shutdown
 neighbor 10.0.0.3 shutdown

Juniper Junos

# Baseline
protocols {
    bgp {
        group internal {
            type internal;
            local-address 10.0.0.1;
            cluster 10.0.0.1;
            neighbor 10.0.0.2 {
                family inet {
                    unicast;
                }
            }
            neighbor 10.0.0.3 {
                family inet {
                    unicast;
                }
            }
        }
    }
}

Isolation:

# Disable the whole BGP group
deactivate protocols bgp group internal
commit

Or to keep the RR up but stop reflecting:

set protocols bgp group internal neighbor 10.0.0.2 family inet unicast route-reflector-client disable
set protocols bgp group internal neighbor 10.0.0.3 family inet unicast route-reflector-client disable
commit

Linux BIRD 2.x

# /etc/bird/bird.conf
router id 10.0.0.1;

protocol bgp rr_clients {
    local as 65000;
    source address 10.0.0.1;
    import all;
    export all;
    route reflector client;
    neighbor 10.0.0.2 as 65000;
    neighbor 10.0.0.3 as 65000;
}

Isolation:

birdc disable all   # stops BIRD entirely

Or to stop reflecting while keeping sessions up:

birdc configure <<EOF
protocol bgp rr_clients {
    route reflector client disable;
}
EOF

Implementing Traffic Generation and Monitoring Tools

A reproducible experiment needs deterministic traffic and telemetry.

Traffic generator (TRex) – simple UDP flow:

# Start TRex in stateless mode
sudo ./t-rex-64 -i -f cap2/dns.yaml   # generates 1 Gbps UDP payload

Monitoring stack:

Python poller (convergence & POR):

#!/usr/bin/env python3
import time, subprocess, json

def get_bgp_summary():
    out = subprocess.check_output(["show", "bgp", "summary"], text=True)
    return sum(1 for line in out.splitlines() if "Established" in line)

def get_rib_prefixes():
    out = subprocess.check_output(["show", "ip", "route", "bgp"], text=True)
    return sum(1 for line in out.splitlines() if line and not line.startswith("Codes"))

def get_fib_prefixes():
    out = subprocess.check_output(["show", "ip", "route"], text=True)
    return sum(1 for line in out.splitlines() if line and not line.startswith("Codes"))

def compute_por():
    rib = get_rib_prefixes()
    fib = get_fib_prefixes()
    return fib / rib if rib else 0.0

def wait_for_convergence(timeout=60, expected_peers=4):
    start = time.time()
    while time.time() - start < timeout:
        est = get_bgp_summary()
        if est == expected_peers:
            rib = get_rib_prefixes()
            fib = get_fib_prefixes()
            if rib == fib:
                return time.time() - start
        time.sleep(1)
    return None   # timeout

if __name__ == "__main__":
    print("Waiting for convergence...")
    conv_time = wait_for_convergence()
    print(f"Convergence time: {conv_time:.2f}s" if conv_time else "Timeout")
    print(f"POR: {compute_por():.2%}")

The poller runs continuously; the convergence time is recorded when the BGP session count matches the expected number and the RIB equals the FIB (no stale routes).

Stop Conditions and Convergence Budgets

Defining Stop Conditions for the Experiment

The experiment terminates when any of the following occurs:

1. Convergence achieved – All BGP sessions in Established state, RIB equals FIB, and no UPDATE messages observed for a quiet period of 30 seconds.
2. Hold‑timer expiry – Any peer transitions to Idle or Active due to Hold Timer Expired (indicates a potential cascade).
3. Resource exhaustion – CPU > 90 % for > 10 s, or BGP process memory > 95 % of allocated.
4. Maximum elapsed time – Hard ceiling of 10 minutes from isolation injection (prevents endless runs).

The first condition is the success criterion; the latter three represent failure or unexpected outcomes that trigger immediate data capture and experiment halt.

Establishing Measurable Convergence Budgets

From the baseline we derive quantitative budgets: