Introduction to FRR and Linux Lab Setup

FRR (Free Range Routing) runs routing protocols as user‑space daemons that update the Linux kernel FIB through Netlink sockets. Zebra maintains the RIB and pushes changes to the kernel. Even when Zebra reports convergence and BGP peers are established, the kernel FIB can stay stale if Netlink message delivery or processing order is disrupted, creating a black‑hole despite clean control‑plane output.

Lab Environment Requirements

• Linux host with kernel ≥5.10 (for nlmon and modern Netlink diagnostics)
• ≥2 GB RAM and 2 vCPU (scale for larger BGP tables)
• Docker Engine ≥20.10 or a bare‑metal VM for containerlab
• Containerlab ≥0.30
• FRR ≥8.0 (includes improved Netlink error handling)
• Optional tools: tcpdump, iproute2, gdb, strace, perf

Topology Definition (clab.yml)

name: frr-netlink-lab
topology:
  nodes:
    r1:
      kind: linux
      image: frrouting/frr:8.4
      cmd: /sbin/init
      binds:
        - ./frr.conf:/etc/frr/frr.conf
        - ./daemons:/etc/frr/daemons
    r2:
      kind: linux
      image: frrouting/frr:8.4
      cmd: /sbin/init
      binds:
        - ./frr.conf:/etc/frr/frr.conf
        - ./daemons:/etc/frr/daemons
  links:
    - endpoints: ["r1:eth1", "r2:eth1"]

Deploy and access:

clab deploy -t clab.yml
clab console frr-netlink-lab r1   # repeat for r2 as needed

Installing FRR on Host (or Inside Container)

# Debian/Ubuntu
curl -s https://deb.frrouting.org/frr/keys.asc | sudo apt-key add -
echo 'deb https://deb.frrouting.org/frr stable main' | sudo tee /etc/apt/sources.list.d/frr.list
sudo apt-get update && sudo apt-get install -y frr frr-pythontools

sudo systemctl enable frr
sudo systemctl start frr

# Verify
sudo vtysh -c "show version"
sudo vtysh -c "show summary"

Ensure zebra, bgpd, and staticd are enabled in /etc/frr/daemons.

Understanding Zebra, BGP, and Netlink

Zebra and Kernel FIB Synchronization

Zebra holds a RIB in memory. On a route change it builds a Netlink RTM_NEWROUTE or RTM_DELROUTE message and sends it via a NETLINK_ROUTE socket. The kernel replies with NLMSG_ERROR or NLMSG_ACK. Zebra treats the absence of an error as success. If the kernel’s Netlink receive queue overflows, messages may be dropped silently, leaving the FIB unchanged while Zebra believes the update succeeded.

BGP Convergence and Route Installation

BGP peers exchange UPDATE messages; after best‑path selection, bgpd informs zebra via the Zserv API. Zebra then installs the route. Convergence is declared when:

• All BGP peers are in Established state (show ip bgp summary)
• Zebra reports no pending RIB operations (show ip rib)
• BGP table matches RIB count (show ip route bgp vs show ip route)

If Zebra’s RIB updates but the kernel FIB lags, traffic follows old paths or is dropped.

Netlink Protocol and Kernel Interaction

Netlink is a datagram‑based, reliable‑ish IPC mechanism:

• Messages carry a sequence number (nlmsg_seq) and may be multi‑part (NLMSG_DONE).
• Each socket has a receive queue sized by net.core.rmem_default and net.core.rmem_max.
• Overflows increment the socket’s rx_drops counter, visible in /proc/net/netlink.
• The kernel processes Netlink asynchronously; under heavy load processing may be deferred, causing temporary FIB staleness.

Troubleshooting Stale Kernel FIB Issues

Identifying Symptoms

1. Control‑plane looks clean: show ip bgp shows routes, show ip route matches RIB.
2. Data‑plane failure: Ping/traceroute to a destination via the new path fails or loops.
3. Packet counters: ip -s link show ethX shows increased RX dropped or TX errors on the egress interface.
4. Netlink diagnostics: Rising rx_drops in /proc/net/netlink for the Zebra socket.
5. Kernel logs: Occasional NETDEV WATCHDOG or netlink: … dropped messages in dmesg.

Analyzing Zebra and BGP Logs

Enable verbose logging temporarily:

sudo vtysh -c "debug zebra events"
sudo vtysh -c "debug bgp updates"
sudo vtysh -c "log file /var/log/frr/frr-debug.log"

Reproduce the scenario (e.g., withdraw a prefix) and look for:

• Zebra log: ZEBRA: Installing route <prefix> via <nexthop> followed by ZEBRA: Netlink send succeeded.
• No subsequent ZEBRA: Netlink recv error or ZEBRA: Route delete.
• BGP log shows the UPDATE processed and best‑path changed.

If Zebra logs a successful Netlink send but the route never appears in ip route show, suspect a Netlink drop.

Checking Netlink Backlog and Install Ordering

List Netlink sockets and queue lengths:

cat /proc/net/netlink

Sample output:

sk       Eth Pid    Groups   Rmem   Wmem   Dump   Locks   Drop   Inode
ffff... 0   1234   0        0      0      0      0       0      12345
ffff... 0   5678   0        65536  65536  0      0       57     67890

The Drop column shows packets dropped due to a full receive queue. Correlate the PID with the Zebra process:

pidof zebra
grep "$(pidof zebra)" /proc/net/netlink

If Drop is non‑zero and increasing, the kernel is discarding Zebra’s messages.

Check socket buffer sizes:

sysctl net.core.rmem_default net.core.rmem_max

If the observed rmem is far below the burst size of Zebra’s Netlink bursts, increase it.

Code and CLI Examples for Troubleshooting

Capturing Netlink Traffic with tcpdump

sudo ip link add nlmon0 type nlmon
sudo ip link set nlmon0 up
sudo tcpdump -i nlmon0 -nn -s 0 -vv -w zebra-netlink.pcap

In another terminal, trigger a route change (e.g., enable zebra debug then withdraw a prefix). Stop capture after a few seconds and inspect:

tcpdump -r zebra-netlink.pcap -nn -vv

Look for RTM_NEWROUTE messages with incrementing nlmsg_seq. Missing NLMSG_ACK or NLMSG_ERROR indicates a silent drop.

FRR‑Specific Debug Commands

vtysh -c "show bgp neighbors"
vtysh -c "show ip route bgp"
vtysh -c "show ip rib"
vtysh -c "show ip rib detail"

Bash Script to Poll RIB vs FIB

#!/usr/bin/env bash
INTERVAL=5
THRESHOLD=10   # max allowed diff before alert

while true; do
    zebra_count=$(vtysh -c "show ip route" | grep -E '^[^ ]' | wc -l)
    kernel_count=$(ip -4 route show | grep -E '^[^ ]' | wc -l)
    diff=$((zebra_count - kernel_count))
    if (( diff > THRESHOLD )); then
        echo "$(date): WARNING: Zebra RIB $zebra_count vs kernel FIB $kernel_count (diff=$diff)"
        grep "$(pidof zebra)" /proc/net/netlink
    fi
    sleep $INTERVAL
done

Run the script in the background; monitor syslog or console for alerts.

Scaling Limitations and Performance Considerations

Impact of High BGP Route Counts on Zebra

Zebra stores each route as a struct rib_entry (~200 bytes per IPv4 route). At 2 million routes, expect >400 MiB for the RIB plus overhead for nexthop and attribute structures. CPU spikes occur during bulk updates because Zebra must:

1. Walk the RIB to delete old paths.
2. Build Netlink messages (often batched, but each route still generates a separate RTM_NEWROUTE).
3. Wait for kernel acknowledgments (blocking if the socket buffer fills).

Netlink Backlog and Performance Optimization

Key knobs:

• Increase socket receive buffer: sysctl -w net.core.rmem_max=268435456 (256 MiB)
• Adjust Netlink max backlog: sysctl -w net.core.netdev_max_backlog=5000 (affects device queues, indirectly influencing Netlink softirq)
• Tune Zebra’s Netlink batch size: FRR 8.0+ supports rib batch-size <num> in zebra config (default 100). Raising to 500 reduces syscall frequency but increases per‑batch latency.
• Enable net.core.optmem_max for larger ancillary data buffers if using timestamping.

Persist settings via /etc/sysctl.d/99-frr-netlink.conf:

net.core.rmem_max=268435456
net.core.rmem_default=16777216
net.core.netdev_max_backlog=4096

Apply with sysctl --system.

Scaling FRR and Linux for Large‑Scale Networks

• Shard Zebra instances: Run multiple Zebra daemons each managing a subset of the RIB (via VRF or table ID). FRR supports named instances (zebra1, zebra2) each with its own socket.
• Use FRR kernel‑mode forwarding (linux kernel dataplane) for pure forwarding, keeping Zebra only for control plane.
• Offload BGP to hardware (if available) and retain Zebra for RIB sync only.
• Monitor with Prometheus + frr_exporter; alert on frr_zebra_netlink_drops_total rising.

Advanced Troubleshooting Techniques

Tuning Kernel Parameters with sysctl

Beyond socket buffers, adjust routing GC thresholds to prevent premature route flushes that can exacerbate apparent staleness:

sysctl -w net.ipv4.route.gc_thresh=2097152
sysctl -w net.ipv4.route.max_size=2097152

Persist changes in a sysctl file and reload with sysctl --system.

Using perf to Trace Netlink Processing

sudo perf trace -e netlink* -p $(pidof zebra)

Watch for dropped or delayed Netlink events during route churn.

Leveraging FRR’s Built‑in Metrics

Enable the Prometheus exporter:

sudo systemctl enable frr_exporter
sudo systemctl start frr_exporter

Then query:

frr_zebra_netlink_drops_total
frr_zebra_rib_routes
frr_bgp_routes

A rising drops_total concurrent with stable RIB counts points to Netlink backlog issues.

This guide provides a reproducible lab, systematic troubleshooting steps, and scaling advice to diagnose and mitigate the scenario where Zebra reports convergence but the kernel FIB remains stale, causing traffic black‑holes.