The template said eight queues but the host had two

Introduction to Multiqueue Dataplane

A multiqueue dataplane exposes multiple independent transmit (TX) and receive (RX) queues to the OS stack. Each queue is serviced by a separate NAPI poll loop or interrupt context, enabling parallel packet processing across CPU cores. In a typical Linux host the dataplane consists of:

• NIC hardware queues (ethtool -l)
• Kernel network device queues (numtxqueues/numrxqueues sysfs attributes)
• Virtio‑net queues in a guest (QEMU -netdev virtio-net-pci,queues=N,...)
• Optional vhost‑user or DPDK queues when userspace accelerators are used

The kernel distributes incoming packets via Receive Side Scaling (RSS) (or similar flow‑distribution) and transmits by selecting a queue based on the socket’s TX hash or SO_INCOMING_CPU affinity.

Importance of Fan‑out

Fan‑out spreads flows evenly across all available queues. Without it:

• One queue becomes a hotspot while others idle → under‑utilized CPU cores.
• Increased tail latency and packet loss during bursts because the single queue’s NAPI budget is exhausted.
• Interrupt storms if the hot queue’s interrupt line is not affined correctly.

Effective fan‑out requires alignment at three layers:

1. Intended topology – design‑specified queue count per interface.
2. Rendered configuration – actual OS/device settings after provisioning.
3. Live queue state – runtime observation of queue usage, interrupt distribution, and packet counters.

A mismatch anywhere collapses the multiqueue benefit.

Understanding Intended Topology Data

Retrieving Intended Topology Data

Intended topology is stored in a source‑of‑truth system (NetBox, Nautobot, or a YAML inventory) and rendered by a templating engine (Jinja2, GoTpl) or Containerlab. Example Containerlab topology (clab-multiqueue.yml):

name: clab-multiqueue
topology:
  nodes:
    host1:
      kind: linux
      image: ubuntu:22.04
      exec:
        - apt-get update && apt-get install -y ethtool iproute2
      vars:
        intf_queue_count: 4
    guest1:
      kind: linux
      image: ubuntu:22.04
      exec:
        - apt-get update && apt-get install -y qemu-system-x86 qemu-utils
      vars:
        intf_queue_count: 4
  links:
    - endpoints: ["host1:eth1", "guest1:eth1"]

vars.intf_queue_count expresses the intended fan‑out: each endpoint should expose four TX and four RX queues.

Analyzing Intended Topology Data for Fan‑out Configuration

From the intended data we derive a target configuration script. A simple Jinja2 template for the host side:

{# host-network-config.j2 #}
{% for iface in interfaces %}
# Set {{ iface.name }} to {{ host.vars.intf_queue_count }} queues
ethtool -L {{ iface.name }} combined {{ host.vars.intf_queue_count }}
ip link set dev {{ iface.name }} numtxqueues {{ host.vars.intf_queue_count }} numrxqueues {{ host.vars.intf_queue_count }}
{% endfor %}

Rendering this template with the intended topology yields the exact CLI commands that should be executed on each node. Any deviation between the rendered script and the live system indicates configuration drift.

Rendered Configs Analysis

Obtaining Rendered Configs

After Containerlab creates the lab, each node’s initialization scripts are stored under clab/ (or can be retrieved via docker exec). For the host node:

docker exec -it clab-multiqueue-host1 cat /etc/network/if-up.d/multiqueue.sh

Sample rendered script (multiqueue.sh):

#!/bin/bash
ethtool -L eth1 combined 4
ip link set dev eth1 numtxqueues 4 numrxqueues 4

Comparing Rendered Configs with Intended Topology Data

We compare the rendered script’s queue count (4) with the intended value (intf_queue_count: 4). A diff reveals mismatches:

# Intended value extracted from topology
INTENDED=$(yq e '.topology.nodes.host1.vars.intf_queue_count' clab-multiqueue.yml)
RENDERED=$(docker exec clab-multiqueue-host1 grep -oP 'combined \K\d+' /etc/network/if-up.d/multiqueue.sh)

if [[ "$INTENDED" != "$RENDERED" ]]; then
  echo "Drift: intended=$INTENDED rendered=$RENDERED"
fi

If the output shows a drift (e.g., rendered=1), the fan‑out was not applied.

Identifying Discrepancies in Fan‑out Configuration

Detecting the exact layer where divergence occurs is the first troubleshooting step.

Live Queue State Analysis

Collecting Live Queue State Data

On the host node we gather:

# Number of queues the NIC claims to support
ethtool -l eth1
# Current queue configuration (ring sizes)
ethtool -g eth1
# Actual TX/RX queues in use
cat /sys/class/net/eth1/queues/tx-0/xps_cpus   # repeat for each queue
# Interrupt distribution
cat /proc/interrupts | grep eth1
# Packet counters per queue
ethtool -S eth1 | grep -E 'rx_queue_[0-9]+_packets|tx_queue_[0-9]+_packets'

Inside a guest (if using virtio‑net):

# Inside the guest VM
ethtool -l eth0
ethtool -g eth0
cat /sys/class/net/eth0/queues/rx-0/rps_cpus

Analyzing Live Queue State for Fan‑out Issues

A healthy multiqueue setup shows:

• ethtool -l reports Combined: 4 (or separate TX/RX counts).
• Each queue’s xps_cpus or rps_cpus bitmap is non‑zero and ideally spread across distinct CPUs.
• /proc/interrupts shows roughly equal interrupt counts per queue.
• Packet counters increase on all queues during traffic.

Symptoms of fan‑out failure:

• ethtool -l shows Combined: 1 despite intended 4.
• Only rx-0/tx-0 counters increase; others stay at zero.
• Interrupts concentrate on a single CPU (e.g., eth1-TxRx-0 only).

Correlating Live Queue State with Intended Topology Data and Rendered Configs

Correlation workflow:

1. Check intended → intf_queue_count = 4.
2. Check rendered → script contains ethtool -L eth1 combined 4.
3. Check live → ethtool -l eth1 returns Combined: 1.

If steps 1 and 2 match but step 3 differs, the problem lies between rendered config and live state (e.g., script not executed, or NIC/driver overrides the setting). If step 2 already mismatched, the error is in the rendering/templating stage.

Troubleshooting Fan‑out Issues

Identifying Host‑Side Issues

1. Script execution failure
   Check logs: journalctl -u systemd-networkd or dmesg for errors from ethtool.
   Fix: Ensure the script has execute permission and is called by the appropriate network manager (e.g., if-up.d, NetworkManager dispatcher, or netplan hooks).

2. ETHTOOL_OPTS ignored
   Some distributions reset queue settings via /etc/network/interfaces or netplan. Verify that no later configuration overwrites the earlier ethtool -L.
   Fix: Place the queue configuration in a /etc/systemd/network/25-eth1.link file:

   [Match]
   Name=eth1
   
   [Link]
   MTUBytes=1500
   # ethtool equivalents
   TCQDiscardOnDrop=0
   # Not directly available; use udev rule or execStartPost

3. Driver/hardware limitation
   Run ethtool -i eth1 to see the driver. Some older NICs (e.g., igb on certain Xeon) expose a maximum of 2 queues regardless of ethtool request.
   Fix: Upgrade firmware, use a NIC with sufficient queues, or fall back to RSS with multiple receive rings via ethtool -L eth1 rx 4 tx 4.

4. IRQ affinity not set
   Even with queues configured, if all interrupts are bound to CPU0, only one NAPI context runs.
   Check: cat /proc/interrupts | grep eth1.
   Fix: Use irqbalance or manually set affinity:

   for i in $(seq 0 3); do
       echo $((1<<$i)) > /proc/irq/$(grep eth1 /proc/interrupts | awk '{print $1}' | cut -d: -f1 | sed -n "$((i+1))p")/smp_affinity_list
   done

Identifying Guest‑Side Issues

1. Virtio‑net queue count not negotiated
   QEMU command line missing queues=N or using outdated virtio version.
   Check inside guest: ethtool -l eth0 shows 1 queue.
   Fix: Adjust QEMU launch:

   -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:56,queues=4,mq=on,vectors=6

   (vectors = queues + 1 for the config vector).

2. Guest driver lacking multiqueue support
   Older kernels (< 3.8) or disabled VIRTIO_NET_CTRL_MQ feature.
   Check: dmesg | grep -i virtio for Virtio net: feature 0x...
   Fix: Upgrade guest kernel or enable the feature via QEMU -device virtio-net-pci,packed=on,mergeable=on.

3. vhost‑user queue mismatch
   When using DPDK vhost‑user, the vhost socket must be created with the same queue number as the NIC.
   Check: vhost_user_get_queue_num in the DPDK app logs.
   Fix: Ensure the --socket-mem and --rxq/--txq arguments match the intended fan‑out.

Common Fan‑out Configuration Mistakes

By systematically verifying intended topology, rendered configuration, and live queue state—and applying the fixes above—you can pinpoint why a supposedly multiqueue dataplane collapses under bursts and restore proper fan‑out.