It Was Not the Forwarder, It Was CPU Throttling

Introduction to Dataplane Regression

Defining Dataplane Regression

A dataplane regression is a measurable decline in the forwarding performance of a Linux‑based network element (e.g., a vRouter, a container‑based switch, or a DPDK‑accelerated NIC) that cannot be explained by a change in the intended control‑plane configuration. The regression appears as lower throughput, higher latency, or increased packet loss while the software version, topology, and traffic matrix remain unchanged.

Identifying Symptoms of Dataplane Regression

When any of these appear without an obvious control‑plane change, the troubleshooting process must consider administrative limits (cgroup, cpuset, scheduler) before assuming an architectural bottleneck (e.g., NIC datapath, lock contention, or inefficient forwarding code).

Troubleshooting Methodology

Initial Assessment and Data Collection

1. Baseline the environment – capture kernel version, Containerlab topology, FRR version, NIC driver, and ethtool settings.

   uname -r
   containerlab --version
   frr -v
   ethtool -i eth0
   sysctl -a | grep -E 'net.core|net.ipv4|vm.'

2. Collect a reproducible traffic test – a steady‑state iperf3 server/client pair running for 30 s, logging both throughput and CPU utilization.

   # server
   iperf3 -s -1 -i 1 > server.log &
   # client (run on another node)
   iperf3 -c <server_ip> -t 30 -i 1 -P 4 > client.log

3. Gather system‑wide telemetry – pidstat, vmstat, iostat, and per‑interface drop counters. Store timestamps for correlation.

   pidstat -p ALL 1 > pidstat.log &
   vmstat 1 > vmstat.log &
   iostat -x 1 > iostat.log &

4. Check for recent changes – look at journalctl -u containerlab, git diff of lab definition, or any sysctl modifications applied via systemd‑tmpfiles.

If the baseline shows a repeatable drop in forwarding performance, proceed to test the four competing explanations.

Competing Explanations for Dataplane Regression

The troubleshooting journey proceeds by falsifying each hypothesis with targeted observations before moving to the next. If all four are eliminated, the remaining cause is likely an architectural datapath limit.

Investigating Cgroup Quota Limitations

Understanding Cgroup Quota Mechanisms

Linux cgroup v2 exposes two primary bandwidth controllers:

• CPU – cpu.max = $MAX $PERIOD (e.g., 50000 100000 → 50 % of a CPU).
• Memory – memory.max = hard limit in bytes; exceeding it triggers OOM kill or throttling if memory.low is set.

When a process is placed in a non‑root slice (e.g., containerlab.slice), the quotas are inherited unless overridden.

Identifying Quota‑Related Bottlenecks

Symptom to look for: CPU utilization of the dataplane process never exceeds the quota percentage, even when the system shows idle cores. Memory stalls appear as rising pgmajfault or kswapd activity.

CLI Examples for Quota Inspection

# Find the cgroup slice that Containerlab uses for its containers
systemd-cgls | grep containerlab

# Example output: containerlab.slice/containerlab-<labname>.slice
SLICE=$(systemd-cgls --no-legend -o name | grep containerlab | head -n1)

# Show CPU quota
cat /sys/fs/cgroup/$SLICE/cpu.max
# Show memory quota
cat /sys/fs/cgroup/$SLICE/memory.max

# Verify the process is inside the slice
pgrep -f frr | xargs -I{} cat /proc/{}/cgroup | grep $SLICE

If cpu.max reads 50000 100000 (i.e., 50 % of a CPU) and iperf3 never exceeds ~50 % of a single core, the quota is a candidate.

Code Examples for Quota Adjustment

A quick test: temporarily raise the quota to see if throughput improves.

# Raise CPU quota to 200 % (two cores) for 10 s, then revert
echo "200000 100000" | sudo tee /sys/fs/cgroup/$SLICE/cpu.max
sleep 10
# Restore original (capture original first)
ORIG=$(cat /sys/fs/cgroup/$SLICE/cpu.max.bak 2>/dev/null || echo "50000 100000")
echo "$ORIG" | sudo tee /sys/fs/cgroup/$SLICE/cpu.max

Interpretation: If throughput scales linearly with the quota increase, the regression is quota‑limited. If there is no change, move on to the next hypothesis.

Analyzing Cpuset Placement and Isolation

Understanding Cpuset Concepts and Configuration

Cpuset (cgroup v2 subsystem) pins tasks to specific CPUs and memory nodes via:

• cpuset.cpus – a comma‑separated list or range of CPU IDs.
• cpuset.mems – list of NUMA node IDs.

Misplacement can cause:

• Cache cold starts when a thread migrates to a core with a different L3 slice.
• Remote memory accesses if the process runs on a node without local memory, increasing latency.
• CPU oversubscription if many containers share the same isolated set.

Evaluating Cpuset Placement Strategies

Symptom to look for: pidstat -p <pid> 1 shows the dataplane threads bouncing between cores, or numastat shows high numa_miss counters. Also, cat /proc/<pid>/status contains Cpus_allowed_list that does not match the expected isolated set.

CLI Examples for Cpuset Management

# Identify the cpuset of the FRR process (replace <pid>)
PID=$(pgrep -f frr)
cat /proc/$PID/status | grep Cpus_allowed_list
cat /proc/$PID/status | grep Mems_allowed_list

# Or via cgroup filesystem
cat /sys/fs/cgroup/$SLICE/cpuset.cpus
cat /sys/fs/cgroup/$SLICE/cpuset.mems

If cpuset.cpus is 0-3 but the system has 16 cores and the other cores are idle, the dataplane is artificially confined.

Code Examples for Cpuset Optimization

A test: expand the cpuset to all online CPUs and observe the effect.

# Backup original
cp /sys/fs/cgroup/$SLICE/cpuset.cpus /tmp/cpuset.cpus.bak
cp /sys/fs/cgroup/$SLICE/cpuset.mems /tmp/cpuset.mems.bak

# Allow all CPUs and memory nodes
echo 0-$(($(nproc --all)-1)) | sudo tee /sys/fs/cgroup/$SLICE/cpuset.cpus
echo 0-$(($(numactl --hardware | grep 'node(s):' | awk '{print $2}')-1)) | sudo tee /sys/fs/cgroup/$SLICE/cpuset.mems

# Run traffic test again
# ... (run iperf3 as before) ...

# Restore
cat /tmp/cpuset.cpus.bak | sudo tee /sys/fs/cgroup/$SLICE/cpuset.cpus
cat /tmp/cpuset.mems.bak | sudo tee /sys/fs/cgroup/$SLICE/cpuset.mems

Interpretation: If throughput jumps when the cpuset is widened, the original placement was the bottleneck. If no change, the issue lies elsewhere.

Examining Scheduler Throttling and Priority

Understanding Scheduler Algorithms and Configurations

Linux offers two relevant throttling mechanisms:

1. CFS bandwidth control – cpu.max (quota/period) as described earlier. When the quota is exhausted within a period, the scheduler puts the task in a throttled state until the next period.
2. Real‑time (RT) throttling – governed by /proc/sys/kernel/sched_rt_runtime_us (default 950000 µs, i.e., 95 % of CPU time for RT tasks). If RT tasks consume >95 %, the kernel throttles them to prevent starvation of non‑RT tasks.

Additionally, schedutil or sched_rt_period_us can affect how quickly the scheduler re‑evaluates throttling.

Identifying Scheduler-Related Bottlenecks

Symptom to look for:

• dmesg contains sched: RT throttling activated or CPU#: core#: CFS throttled.
• pidstat -p <pid> 1 shows the process spending a large fraction of time in state D (uninterruptible sleep) or S (sleep) while the runqueue length stays high.
• cat /proc/sched_debug (if CONFIG_SCHED_DEBUG enabled) shows high nr_throttled counts for the task’s cgroup.

CLI Examples for Scheduler Inspection

# Check RT throttling settings
cat /proc/sys/kernel/sched_rt_runtime_us
cat /proc/sys/kernel/sched_rt_period_us

# Check CFS throttling for the slice
cat /sys/fs/cgroup/$SLICE/cpu.max   # quota/period
# If quota is low, the effective throttling is visible here

# Look at throttling counters (requires debugfs mounted)
if [ -d /sys/kernel/debug ]; then
    cat /sys/kernel/debug/sched/cpuacct.stat.$SLICE 2>/dev/null || echo "debugfs not available"
fi

# Quick view of scheduler state for the FRR PID
pidstat -p $(pgrep -f frr) 1

If you see nr_throttled increasing steadily while the CPU quota is not exhausted, RT throttling may be the cause.

Code Examples for Scheduler Adjustment

Test 1 – Raise CFS quota (same as cgroup test).
Test 2 – Increase RT runtime to see if RT tasks (if any) were being throttled.

# Backup original RT runtime
ORIG_RT=$(cat /proc/sys/kernel/sched_rt_runtime_us)
# Give RT tasks 100 % of CPU (disable throttling)
echo 1000000 | sudo tee /proc/sys/kernel/sched_rt_runtime_us

# Run traffic test
# ... (iperf3) ...

# Restore
echo $ORIG_RT | sudo tee /proc/sys/kernel/sched_rt_runtime_us

Interpretation: If throughput improves after raising the RT runtime, the regression was due to RT throttling (perhaps FRR spawned a high‑priority watchdog thread). If no change, move on.

Assessing Actual Packet Path Cost and Network Overhead

Understanding Network Architecture and Packet Flow

In a Containerlab‑based FRR router, the datapath typically follows:

1. NIC driver → receives packets into a ring buffer.
2. XDP/eBPF (if loaded) → optional early drop or redirect.
3. Kernel networking stack → netif_receive_skb, ip_rcv, tcp_v4_rcv, etc.
4. FRR’s ZEBRA or specific protocol daemon (e.g., bgpd) may receive packets via netlink or AF_PACKET sockets for control plane; the forwarding plane stays in the kernel.
5. Transmit → dev_queue_xmit, possible tc egress qdisc, then NIC TX ring.

Potential sources of overhead:

• RX/TX ring size too small → drops under burst.
• Excessive tc qdisc (e.g., htb with many classes) adds per‑packet lookup cost.
• GRO/LRO mis‑configuration → either too much CPU (if disabled) or loss of latency (if enabled).
• Lock contention in net_core.dev_weight or netdev_budget.
• CPU cache misses due to data structure walks (e.g., fib lookup).

Evaluating Packet Path Cost and Optimization Strategies

Symptom to look for:

• ethtool -S <if> shows rising rx_drop, tx_drop, or rx_over_errors.
• tc -s qdisc show dev <if> reveals high backlog or drops on egress qdiscs.
• perf record -g -p $(pgrep -f frr) followed by perf report highlights hot spots in the networking stack (e.g., ip_rcv, __dev_queue_xmit).

CLI Examples for Packet‑Path Inspection

# Check NIC ring sizes
ethtool -g eth0

# View current qdisc configuration
tc -s qdisc show dev eth0

# Monitor drops in real time
watch -n 1 "ethtool -S eth0 | grep -E 'rx_drop|tx_drop'"

# Capture perf data for the FRR process (run during iperf test)
perf record -g -p $(pgrep -f frr) -- sleep 30