Graceful restart made the control plane lie

Introduction to Graceful Restart

Graceful Restart (GR) is a control‑plane mechanism (RFC 4724 for BGP, RFC 3623 for OSPFv2/v3, and draft‑ietf-isis-graceful-restart for IS‑IS) that lets a router keep forwarding information for routes learned from a neighbor while that neighbor’s control plane is temporarily unavailable (e.g., during a software upgrade or process restart).

Benefits

• Reduced traffic loss – forwarding entries stay valid, avoiding black‑holes or micro‑loops.
• Stable control‑plane – peers do not withdraw/re‑advertise routes, limiting UPDATE/LSA flooding.
• Operational simplicity – maintenance windows can be shorter because full reconvergence is avoided.
• Predictable behavior – standardized timers and state machines let operators calculate worst‑case disruption windows.

Sequence of Events During Graceful Restart

1. Initial Route Preservation

When a GR‑capable router detects that a neighbor has entered the GR state (signaled by the GR capability in the OPEN message):

1. Marks all routes received from that neighbor as stale (BGP adds the Stale flag; OSPF/IS‑IS retains LSAs/LSPs with a Grace TLV).
2. Preserves the associated forwarding entries (RIB → FIB) unchanged, even though the control plane no longer receives fresh updates.
3. Starts the Restart Timer (restart‑time). While this timer runs, the router expects the neighbor to re‑establish the session and resend its routing information.
4. If the neighbor reconnects before the timer expires, it sends an End‑of‑RIB marker; the helper router removes the Stale flag and treats the routes as fresh.

Result: Forwarding continues uninterrupted while the control plane rebuilds its view.

2. Next‑Hop Withdrawal and Stale Forwarding

If the neighbor’s next‑hop becomes unreachable before the Restart Timer expires:

• Control‑plane withdrawal – the neighbor (or an intermediate router) sends a UPDATE/Withdrawal for the next‑hop prefix, or the IGP declares the next‑hop unreachable (OSPF LS‑Age max, IS‑IS LSP purge). The helper router processes this as a normal withdrawal for the next‑hop only.
• Data‑plane persistence – because the original route remains marked stale, the FIB entry still points to the now‑withdrawn next‑hop. Forwarding continues toward the stale next‑hop until the Stale Path Timer (stalepath‑time) expires.

Result: A mismatch appears— the RIB shows a valid (stale) route, but the FIB’s next‑hop reference is dangling.

3. Dataplane‑Only Failure Window

The interval between the moment the next‑hop is withdrawn and the moment the stale path timer expires is a dataplane‑only failure window:

• Operator view (CLI/dashboard) – show ip route or show ip bgp still lists the prefix as present, often with a Stale tag, suggesting the route is healthy.
• Forwarding view – packets for the prefix are forwarded to the stale next‑hop, which may be unreachable (black‑hole) or misrouted (micro‑loop/sub‑optimal path).
• The window ends when the stale path timer fires; the helper router then removes the stale route from both RIB and FIB, triggering normal reconvergence (if an alternative path exists) or a clean withdrawal.

Why it’s invisible: Standard routing‑table monitors only inspect the RIB/control‑plane state. Dataplane verification (ping, traceroute, flow sampling) is required to observe the symptom.

Troubleshooting Graceful Restart Issues

Identifying Stale Forwarding

1. Look for the Stale flag on suspect prefixes (show ip bgp <prefix> shows Stale; show ip ospf database shows LSAs with Grace TLV).
2. Verify next‑hop reachability:
   • show ip route <next-hop> → “% Subnet not in table” if missing.
   • show ip cef <next-hop> → drop or punt if unresolved.
3. Observe traffic loss with ping/traceroute or inline telemetry (NetFlow, sFlow, IPFIX).
4. Confirm the stale path timer is still running (show ip bgp neighbors <neighbor> | include stale-path-time).

Analyzing Route Tables and Next‑Hop Information

• RIB inspection: show ip route vrf <vrf> <prefix> displays origin, metric, and stale status.
• FIB inspection: show ip cef vrf <vrf> <prefix> detail shows the exact next‑hop used for forwarding and any recursive resolution state.
• Next‑hop recursion: Examine the chain (show ip route <next-hop>) to see if the recursion point is also stale.
• BGP attributes: show ip bgp <prefix> reveals the Next Hop attribute; compare it with the IGP‑reachable set.

Using CLI Commands for Debugging

CLI Examples for Monitoring and Debugging

Show Route Commands

# Cisco IOS
show ip route 10.0.0.0/8
# Sample output: S*   10.0.0.0/8 [1/0] via 192.168.1.2, 00:05:12, Stale

# Juniper
show route 10.0.0.0/8 extensive
# Look for: State: <Active Int Ext> ... Stale

# Linux FRR
show ip route 10.0.0.0/8
# Sample output: S>* 10.0.0.0/8 [200/0] via 10.1.2.3, eth0, weight 1, stale

Show Next‑Hop Commands

# Cisco IOS – check next‑hop reachability
show ip route 192.168.1.2
# If missing: % Subnet not in table

# Cisco IOS – FIB detail
show ip cef 10.0.0.0/8 detail
# Output:
# 10.0.0.0/8, epoch 0, flags[none]
#   recursive via 192.168.1.2
#   [dropped]   <-- indicates unresolved next‑hop

# Juniper
show route forwarding-table destination 10.0.0.0/8
# Displays next‑hop: 192.168.1.2 (unreachable) or drop

# Linux FRR
show ip fib 10.0.0.0/8
# May show: via 10.1.2.3 dev eth0  src 10.0.0.1  metric 0  pref 255
# or:   via 10.1.2.3 dev eth0  src 10.0.0.1  metric 0  pref 255  dead

Debugging Commands for Stale Forwarding

# Cisco IOS – monitor GR state changes
debug ip bgp events
# Look for:
#   %BGP-5-ADJCHANGE: neighbor 10.0.0.1 Down BGP Notification sent
#   %BGP-5-GR: Entering Graceful Restart state for neighbor 10.0.0.1
#   %BGP-5-GR: Stale-path timer expired for neighbor 10.0.0.1

# Juniper – trace GR
monitor start bgp graceful-restart
# Shows helper entering/stale‑path timer start/stop.

# Linux FRR – enable debug
debug bgp events
# Output similar to Cisco.

# Generic – verify traffic loss
ping -c 5 10.0.0.5
# Observe 100% loss while stale next‑hop is down.

traceroute 10.0.0.5
# Shows first hop as the stale next‑hop, then * * * indicating drop.