BQL and TSQ under east-west RPC floods

Introduction to Benchmarking Byte Queue Limits and TCP Small Queues

Overview of Byte Queue Limits and TCP Small Queues

Byte Queue Limits (BQL) and TCP Small Queues (TSQ) are two complementary mechanisms in the Linux networking stack that aim to reduce latency and packet loss caused by excessive buffering.

BQL operates at the network device transmit (TX) queue level. It caps the number of bytes that a driver may hold in its TX ring before signalling the stack to pause further transmission. By limiting the byte depth, BQL prevents the NIC from absorbing large bursts that would otherwise increase queuing delay and increase the chance of hardware drops when the ring overflows.
TSQ works inside the TCP protocol implementation. It places a per‑socket ceiling on the amount of data that may be queued in the TCP write queue (the amount of data awaiting transmission or acknowledgment). When the queue exceeds the TSQ limit, TCP throttles the application’s write calls, thereby keeping the amount of buffered data small and reducing tail latency.

Both controls are designed to fight bufferbloat, but they act at different points in the datapath. Understanding where each control exerts back‑pressure is essential when many short RPC flows converge on a software dataplane (e.g., a userspace proxy, a sidecar, or a kernel‑based load balancer).

Importance of Benchmarking in Software Dataplane

In a software dataplane the CPU, memory subsystem, and queueing disciplines interact closely. A burst of many tiny RPC requests can saturate the TX queues of a virtual NIC, cause the TCP stack to accumulate large write queues, and ultimately lead to retransmits or application‑level timeouts.
Benchmarking BQL and TSQ lets us answer three practical questions:

Where does loss occur? – Is it primarily at the NIC TX ring (hardware drop) or in the TCP stack (application throttling)?
How much does each control shift the backlog? – Does lowering BQL merely move packets from the NIC to the TCP write queue, or does it truly reduce overall buffered bytes?
What is the measurable overhead? – What are the CPU cycles spent in the stack, the increase in system call latency, and the effect on throughput when the limits are tightened?

Answering these questions requires a repeatable benchmark that isolates the datapath, varies the limits, and measures loss, latency, and CPU utilization.

Understanding Byte Queue Limits

Definition and Purpose of Byte Queue Limits

Byte Queue Limits is a per‑TX‑queue attribute introduced to give administrators direct control over the maximum number of bytes that a network driver may queue for transmission. The kernel exports two files under /sys/class/net/<dev>/queues/tx-<n>/:

byte_queue_len – writable; setting this value imposes a hard ceiling on the TX byte count.
byte_queue_limits – read‑only; shows the current limit, the current byte count, and whether the queue is currently throttled.

When the driver attempts to add more bytes than byte_queue_len permits, it returns -EBUSY to the stack, which in turn applies back‑pressure (e.g., by pausing the qdisc or signalling the TCP stack to slow down). The purpose is to bound the amount of memory the NIC can consume and to keep the TX queue short enough that latency‑sensitive traffic does not suffer from large queuing delays.

How Byte Queue Limits Affect Network Performance

Latency: Lowering the BQL reduces the maximum time a packet can sit in the TX ring, decreasing queuing delay.
Burst loss: If a traffic burst exceeds the BQL, the driver will drop packets (or more accurately, refuse to enqueue them) and the stack will see increased transmit timeouts or application‑level stalls.
CPU overhead: A very low BQL can cause the stack to frequently hit the throttling point, leading to more frequent calls into the driver’s start_xmit and increased CPU usage due to repeated retries.
Interaction with offloads: Features like TSO, GSO, and GRO effectively aggregate multiple packets into larger buffers; BQL still counts the byte size after segmentation, so large GSO packets can hit the limit sooner.

Configuring Byte Queue Limits for Optimal Performance

The optimal BQL depends on the NIC’s TX ring size, the expected burst size, and the latency target. A common starting point is to set the limit to roughly one‑third of the TX ring’s byte capacity. For example, if a virtio NIC has a TX ring of 256 descriptors each capable of holding a 2 KB frame, the raw byte capacity is 512 KB; a BQL of 150 KB often yields a good trade‑off.

# Show current TX queue 0 byte queue length and limit
cat /sys/class/net/eth0/queues/tx-0/byte_queue_limits
# Output: max_limit: 65535, min_limit: 1024, current_limit: 4096, current_bytes: 1234

# Set a new limit of 8192 bytes (8 KB) for TX queue 0
echo 8192 > /sys/class/net/eth0/queues/tx-0/byte_queue_len

# Verify the change
cat /sys/class/net/eth0/queues/tx-0/byte_queue_limits

Changes take effect immediately; no reload of the driver is required. When using multiple TX queues (RSS), each queue can be tuned independently.

Understanding TCP Small Queues

Definition and Purpose of TCP Small Queues

TCP Small Queues (TSQ) is a per‑socket flow control mechanism that limits the amount of data that may reside in the TCP write queue (the amount of data awaiting transmission or ACK). The kernel exposes this limit via the sysctl net.ipv4.tcp_limit_output_bytes (soft limit) and, implicitly, a hard limit derived from the socket’s memory allocation. When the write queue exceeds the soft limit, TCP marks the socket as “limited” and returns EAGAIN to the next write() or sendmsg() call, causing the application to back off or retry later.

The purpose of TSQ is twofold:

Reduce latency: By keeping the amount of unacknowledged data small, the RTT variance caused by large buffers shrinks.
Mitigate bufferbloat: Excessive TCP write queues can absorb bursts and hide congestion signals; TSQ forces the application to feel congestion earlier.

How TCP Small Queues Affect Network Performance

Latency: Lower TSQ values decrease the maximum time a byte can spend waiting for transmission, improving tail latency for request/response patterns.
Throughput: If the limit is set below the bandwidth‑delay product (BDP) of the path, the sender will be forced to idle frequently, reducing achievable throughput.
Application impact: Applications that perform large writes (e.g., bulk transfer) will see more EAGAIN returns, leading to increased syscall overhead and potential CPU spin if they busy‑wait.
Interaction with TSQ and BQL: When TSQ is low, the TCP stack will throttle earlier, which reduces the pressure on the NIC TX queues; conversely, a very low BQL can cause the TCP stack to see frequent transmit stalls even if TSQ is high.

Configuring TCP Small Queues for Optimal Performance

The default TSQ limit on most modern kernels is 20 KB (20480 bytes). For latency‑critical RPC workloads, a value in the range of 4–8 KB is often effective. The limit can be changed at runtime:

# View current TSQ soft limit
sysctl net.ipv4.tcp_limit_output_bytes
# net.ipv4.tcp_limit_output_bytes = 20480

# Set a new limit of 4096 bytes (4 KB)
sysctl -w net.ipv4.tcp_limit_output_bytes=4096

# To make it persistent across reboots, add to /etc/sysctl.conf or a file in /etc/sysctl.d/
echo "net.ipv4.tcp_limit_output_bytes=4096" >> /etc/sysctl.d/99-tsq.conf

Because TSQ is per‑socket, the change affects all newly created TCP sockets; existing sockets retain their previous limit until they are closed and reopened.

Benchmarking Methodology

Setting Up the Test Environment

We use two Linux namespaces (or VMs) connected via a virtual Ethernet (veth) pair to emulate a NIC with controllable TX queues. The server side runs a simple TCP echo service that receives a small request, copies it to the response buffer, and sends it back. The client side drives many short RPC flows using the netperf TCP request/response (TCP_RR) test, which mimics a request‑reply RPC pattern.

Host kernel: 5.15+ (any recent kernel with BQL and TSQ).
Server process: nc -l -p 5000 -c 'while read line; do echo "$line"; done' (or a custom C program using send()/recv()).
Client tool: netperf -t TCP_RR -H <server_ip> -l 30 -t 10s (30‑second test, 10‑second warm‑up).
Traffic characteristics: 64‑byte request, 64‑byte response (total 128 bytes) per RPC.

Measurements collected during each run include:

Packet loss at the NIC TX ring (via ethtool -S drop counters).
TCP retransmits and timeout events (from /proc/net/snmp).
Application‑level latency (request‑to‑response time) from netperf.
CPU utilization of the server and client namespaces (via pidstat or perf).
TX queue byte depth and throttling events (from byte_queue_limits).
TCP write queue size (via ss -ti or /proc/<pid>/tcp).

By sweeping BQL (e.g., 2 KB, 8 KB, 32 KB, unlimited) and TSQ (e.g., 2 KB, 4 KB, 8 KB, 20 KB, unlimited) we can observe where loss occurs, how backlog shifts, and the overhead introduced by each control. This methodology yields clear, reproducible insight into the interaction between BQL and TSQ in a software dataplane under bursty RPC workloads.