Rolling next-hop-self without creating hidden blackholes

Introduction

Next‑hop‑self (NHS) and reflection policies are BGP control‑plane knobs that change the next‑hop attribute for routes a router originates or reflects. In IP‑Clos, EVPN‑VXLAN, or segment‑routing fabrics, altering NHS can switch the dataplane path for traffic that traverses the fabric. A mis‑configured NHS/reflection rule can create a control‑plane‑correct but dataplane‑broken state: the RIB looks valid, yet the FIB fails to install the expected next‑hop, causing black‑holes or loops.

Before exposing the entire fabric to a new NHS/reflection setting, each change must be validated at two independent layers:

1. Recursive reachability – the route’s next‑hop resolves via IGP/SR‑TE to an actual outgoing interface.
2. FIB install success – the resolved next‑hop is programmed into the forwarding ASIC or software dataplane.

If either check fails, the change must be halted before it propagates further, limiting blast‑radius to a controlled segment.

Designing the Staged Rollout Process

1. Identify Gateways and Fabric Segments

• Logical segmentation – divide the fabric into change domains (e.g., spine‑leaf pods, aggregation blocks, AS clusters).
• Gateway selection – pick one or two edge devices per domain that will first receive the new NHS/reflection policy. Gateways must:
  • Have full visibility of the IGP/SR‑TE topology (for recursive reachability).
  • Expose FIB stats via telemetry (show fib, show route forwarding-table, etc.).
• Blast‑radius definition – initially limited to the gateway’s immediate peers; expands outward as each gate succeeds.

2. Rollout Schedule and Sequence

Commit boundary – point where configuration is written to running‑config (commit in Junos, write memory in IOS‑XR, etc.).
Verification gate – occurs after the commit but before the change is advertised to peers.

3. Implementing Rollback Mechanisms for Failed Gates

• Operational rollback window – configurable timer (e.g., 90 s) after which the system auto‑reverts to the pre‑change config if verification does not succeed.
• Rollback trigger – any of:
  • Recursive reachability loss (next‑hop unresolved).
  • FIB install failure (missing entry or incorrect next‑hop).
  • BGP session flap caused by the change.
• Rollback action – execute a pre‑saved configuration snapshot (Junos rollback, IOS‑XR replace with rollback file, EOS config replace with rollback). Snapshot taken immediately before the pre‑check and stored in a secure, version‑controlled repo (GitOps).
• Operator intervention point – if automatic rollback fails (device unreachable), on‑call engineer manually issues the rollback command and verifies restoration.

Verifying Recursive Reachability

Understanding Recursive Reachability

Recursive reachability means a route’s next‑hop address is itself reachable via the underlying IGP (or SR‑TE) topology. For BGP, the next‑hop is often a loopback; the IGP must have a valid path to that loopback. If the IGP path disappears, the BGP route becomes unusable even though the BGP table shows it.

Tools and Techniques

Example CLI Commands

Cisco IOS‑XR

# Pre‑check: ensure IGP stability
show ospf neighbor | include Full
show isis neighbor | include Up

# Verify recursive reachability for candidate next-hop (e.g., 10.0.0.1)
show ip route 10.0.0.1

Expected snippet:

Routing entry for 10.0.0.1/32
  Known via "ospf 1", distance 110, metric 20, type intra area
  Last update from 10.0.0.2 via GigabitEthernet0/0/0/1, 00:00:12 ago
  Routing Descriptor Blocks:
  * 10.0.0.2, from 10.0.0.2, via GigabitEthernet0/0/0/1, route-metric 20, traffic share count 1

Juniper Junos

# Pre‑check: IS‑IS adjacency
show isis adjacency | match Up

# Recursive reachability check
show route 10.0.0.1 detail

Look for:

10.0.0.1/32      *[IS-IS/18] 00:00:10, metric 20
                 > to 10.0.0.2 via ge-0/0/1.0

Arista EOS

# Pre‑check: OSPF/EIGRP status
show ip ospf neighbor
show ip eigrp neighbors

# Recursive reachability
show ip route 10.0.0.1

Expected:

O        10.0.0.1/32 [110/20] via 10.0.0.2, Ethernet1

Verifying FIB Install Success

Understanding FIB Installation

The Forwarding Information Base (FIB) is the hardware‑accelerated table used for packet forwarding. A route may reside in the RIB but be absent from the FIB due to TCAM exhaustion, incorrect next‑hop type, or ASIC‑specific programming errors (e.g., missing encapsulation for SR‑TE). Verifying FIB install ensures the control‑plane decision is actually enforced in the dataplane.

Methods for Verification

Automation Examples

Python (Nornir + Netmiko) – IOS‑XR

from nornir import InitNornir
from nornir_netmiko.tasks import netmiko_send_command
from nornir_utils.plugins.functions import print_result

nr = InitNornir(config_file="config.yaml")

def check_fib(task, prefix):
    cmd = f"show fib route {prefix}"
    result = task.run(task=netmiko_send_command, command_string=cmd)
    fib_output = result.result
    if prefix in fib_output and "via" in fib_output:
        task.host["fib_ok"] = True
    else:
        task.host["fib_ok"] = False

gateways = nr.filter(F=F(groups__contains="gateway"))
result = gateways.run(task=check_fib, prefix="10.0.0.0/24")
print_result(result)

if any(not h["fib_ok"] for h in gateways.inventory.hosts.values()):
    raise SystemExit("FIB install failed on one or more gateways")

Ansible Playbook – Junos

- name: Verify FIB install for NHS change
  hosts: gateway
  gather_facts: false
  vars:
    prefix: 10.0.0.0/24
  tasks:
    - name: Retrieve FIB entry
      juniper.junos.junos_command:
        commands:
          - "show route {{ prefix }} detail"
      register: fib_out

    - name: Parse output for active route
      set_fact:
        fib_ok: "{{ fib_out.stdout[0] | regex_search('\\* ' + prefix) }}"

    - name: Fail if FIB missing
      fail:
        msg: "FIB install failed for {{ prefix }} on {{ inventory_hostname }}"
      when: not fib_ok

Troubleshooting Common Issues

Recursive Reachability Problems

FIB Install Failures

End of document.