LinkState
Go back

Diagnosing Next-Hop Recursion Failure Three Ways

Introduction to Next-Hop Recursion Failure

A next-hop recursion failure occurs when a router’s control plane installs a route whose next‑hop address is unresolved because the next‑hop depends on another route that is absent, inactive, or resolves to an unreachable interface. The control plane may appear healthy—RIB entries are present, BGP/OSPF/IS‑IS adjacencies are up—but the FIB cannot program the forwarding entry because the recursive resolution chain terminates in a null or black‑hole next‑hop.

Common causes

Symptoms

Initial troubleshooting

  1. Verify the route entry (show ip route <prefix>).
  2. Check next‑hop reachability (ping <next-hop> or traceroute <next-hop>).
  3. Confirm IGP/BGP adjacency state (show ip ospf neighbor, show bgp summary).
  4. Inspect applied route‑policy (show route-policy <name>).

If the next‑hop is reachable but the route remains inactive, the problem lies in the recursion chain, requiring a deeper look at the forwarding dependency graph.

Direct CLI Tracing

Overview

Direct CLI tracing uses the router’s built‑in diagnostic commands to inspect the RIB, FIB, and protocol tables. Typical commands include:

The operator must correlate the output manually to locate where the recursion breaks.

Step‑by‑Step Procedure

  1. Identify the problematic prefix from alerts or user reports.
  2. Display the RIB entry and note the next‑hop(s). 
    show ip route 10.0.0.0/24
  3. Check each next‑hop for reachability in the RIB/FIB. 
    show ip route <next-hop>
show cef ipv4 <next-hop> detail
  4. If the next‑hop is missing, trace why:
    • Look for IGP advertisement (show ospf database, show isis database).
    • Check BGP attributes (show bgp <next-hop>).
    • Verify any route‑policy that may have filtered or altered the next‑hop.
  5. Repeat steps 2‑4 for each discovered next‑hop until a reachable next‑hop is found or a dead‑end is reached.
  6. Confirm FIB programming for the prefix once a valid next‑hop is resolved. 
    show cef ipv4 10.0.0.0/24 detail
  7. Document the failure point (e.g., “Next‑hop 192.168.1.10 missing from RIB because OSPF LSA Type‑1 not advertised due to passive‑interface on Gig0/1”).

Example CLI Commands

VendorCommandPurpose
Cisco IOS‑XRshow route 10.0.0.0/24 detailRIB entry with next‑hop and source protocol
Cisco IOS‑XRshow cef ipv4 10.0.0.0/24 detailFIB entry and hardware programming status
Cisco IOS‑XRshow bgp 10.0.0.0/24BGP path attributes, next‑hop, and any filtering
Cisco IOS‑XRshow ospf interface GigabitEthernet0/0/0/1Verify interface state for OSPF adjacency
Juniper Junosshow route 10.0.0.0/24 extensiveRIB, next‑hop, and forwarding‑table details
Juniper Junosshow route receive-protocol bgp 10.0.0.0/24BGP‑received route before import policy
Juniper Junosshow ospf neighborOSPF adjacency state
Arista EOSshow ip route 10.0.0.0/24 vrf defaultRIB entry
Arista EOSshow ip bgp 10.0.0.0/24BGP path details
Arista EOSshow ip route <next-hop>Next‑hop reachability

Limitations

Scripted Table Collection

Introduction

Scripted collection automates retrieval of RIB, FIB, and protocol tables from multiple devices, normalizes the output (e.g., JSON or CSV), and enables offline graph construction and analysis. Typical approaches:

The scripted approach decouples data gathering from analysis, allowing reuse of the same dataset for multiple investigations.

Script Examples

Python + Netmiko (Cisco IOS‑XR)

#!/usr/bin/env python3
import json
from netmiko import ConnectHandler

devices = [
    {"host": "r1.example.com", "username": "admin", "password": "****", "device_type": "cisco_xr"},
    {"host": "r2.example.com", "username": "admin", "password": "****", "device_type": "cisco_xr"},
]

def collect_rib_bgp(dev):
    try:
        conn = ConnectHandler(**dev)
        conn.enable()
        rib_out = conn.send_command("show route 10.0.0.0/24 detail", use_textfsm=True)
        bgp_out = conn.send_command("show bgp 10.0.0.0/24", use_textfsm=True)
        conn.disconnect()
        return {
            "host": dev["host"],
            "rib": rib_out,
            "bgp": bgp_out,
        }
    except Exception as e:
        return {"host": dev["host"], "error": str(e)}

results = [collect_rib_bgp(d) for d in devices]
with open("routing_snapshot.json", "w") as f:
    json.dump(results, f, indent=2)

Ansible (Juniper Junos)

- name: Gather routing tables for next‑hop analysis
  hosts: juniper_routers
  gather_facts: false
  tasks:
    - name: Run show route extensive
      juniper_junos_command:
        commands:
          - show route 10.0.0.0/24 extensive
      register: route_out

    - name: Run show bgp path
      juniper_junos_command:
        commands:
          - show bgp 10.0.0.0/24
      register: bgp_out

    - name: Save to file
      copy:
        content: |
          {{ inventory_hostname }}
          ROUTE: {{ route_out.stdout[0] }}
          BGP:   {{ bgp_out.stdout[0] }}
        dest: "/tmp/routing_{{ inventory_hostname }}.txt"

Analyzing Collected Data

Once normalized, a simple graph algorithm can detect missing next‑hops:

import json, sys

def load_snapshot(path):
    with open(path) as f:
        return json.load(f)

def build_rib_map(snapshot):
    rib = {}
    for entry in snapshot:
        host = entry["host"]
        rib[host] = {}
        for route in entry.get("rib", []):
            prefix = route["prefix"]
            nexthops = route.get("nexthop", [])
            rib[host][prefix] = nexthops
    return rib

def find_missing_next_hops(rib_map):
    missing = []
    for host, prefixes in rib_map.items():
        for prefix, nexthops in prefixes.items():
            for nh in nexthops:
                found = any(nh in v for v in rib_map.values())
                if not found:
                    missing.append((host, prefix, nh))
    return missing

if __name__ == "__main__":
    snap = load_snapshot("routing_snapshot.json")
    rib = build_rib_map(snap)
    failures = find_missing_next_hops(rib)
    for host, prefix, nh in failures:
        print(f"[{host}] Prefix {prefix} depends on unresolved next‑hop {nh}")

The script prints each prefix whose next‑hop does not appear as a destination in any collected RIB, pinpointing the recursion break.

Scaling Limitations

AI‑Assisted Tool Orchestration

Overview

AI‑assisted orchestration platforms ingest telemetry (SNMP, syslog, gNMI, streaming telemetry), logs, and occasional on‑demand CLI outputs, then apply machine‑learning models or rule‑based inference engines to surface probable root causes. Examples:

These tools typically output a probability score or a ranked list of hypotheses (e.g., “Next‑hop recursion failure due to missing OSPF LSA for 192.168.10.0/24 – 87% confidence”).

Configuring AI‑Assisted Tools for Recursion Failure Detection

  1. Enable telemetry streams for RIB/FIB and protocol state.
    • Cisco IOS‑XR: telemetry model-driven subscription to Cisco-IOS-XR-ipv4-rib-oper and Cisco-IOS-XR-bgp-oper.
    • Juniper Junos: set system services extension-service request-response grpc and subscribe to junos-routing/rib.
  2. Forward logs (syslog or structured logging) to the AI platform’s ingest endpoint.
  3. Define a detection model (often pre‑built) that looks for:
    • RIB entry present AND Next‑hop address not present in any RIB/FIB AND No active BGP/OSPF/IS‑IS withdrawal for that next‑hop.
  4. Set alert thresholds (e.g., trigger when the condition persists > 30 seconds).
  5. Integrate with ticketing or automation (e.g., create a ServiceNow incident or run a remediation playbook).

Example Use Cases and CLI/Code Snippets

These snippets illustrate how telemetry, logs, and lightweight analytics can be combined to automatically surface next‑hop recursion failures without manual CLI traversal.

Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email
Previous Post
Git as audit trail, SoT as live model
Next Post
What to log from a tool-calling workbench