Scheduler Latency, Cgroup Pressure, and IRQ Affinity: Impact on Burst Absorption

Introduction

Scheduler latency is the delay between a task becoming runnable and the kernel actually scheduling it onto a CPU. High latency adds jitter to time‑sensitive work such as network packet processing, timer expiry, or userspace polling loops.

Cgroup pressure measures how much a control group is contending for a limited resource (CPU, memory, I/O). When a cgroup hits its limit, the kernel throttles or reclaims, increasing wake‑up latency for tasks inside the group and potentially causing missed deadlines.

IRQ affinity determines which CPU(s) handle a given hardware interrupt. If the affinity is mis‑aligned with a CPU already overloaded by scheduler or cgroup pressure, the interrupt handler may be delayed, leading to NIC receive‑ring overruns and packet drops.

Together, these three factors can erode the ability of a fixed‑size queue (e.g., NIC RX ring) to absorb microbursts, even when the queue depth itself remains unchanged.

Interaction

• Scheduler latency → IRQ handling: A softirq (NET_RX) is raised from interrupt context and executed in process context. If the scheduler cannot promptly run the ksoftirqd thread, the softirq backlog grows.
• Cgroup pressure → scheduler latency: CPU‑cgroup throttling adds a deterministic delay to the time a task spends in the runqueue before being selected, inflating scheduler latency for all tasks in that cgroup.
• IRQ affinity → cgroup pressure: Binding an IRQ to a CPU that belongs to a heavily throttled cgroup forces the interrupt handler to run under the same throttling constraints, compounding latency.

Understanding this interaction is essential because tuning one knob in isolation (e.g., raising NIC RX queue length) may yield no improvement if the scheduler or cgroup layer is the true bottleneck.

Queue Depth and Burst Absorption

Configuring Queue Depth

Queue depth for a network interface is the number of descriptors the NIC can hold in its receive (RX) and transmit (TX) rings before it must signal the driver to replenish or reclaim buffers. In Linux, the depth is exposed via ethtool:

# Show current RX/TX ring sizes
ethtool -g eth0

Typical values range from 64 to 4096 descriptors. Raising the depth gives the NIC more room to absorb short bursts of packets before the driver must refill the ring.

Factors Affecting Effective Burst Absorption

Thus, a static queue depth number does not guarantee burst absorption; the effective depth is limited by the slowest stage in the packet path: NIC → interrupt → softirq → scheduler → cgroup → userspace consumer.

Measuring Scheduler Latency

Tools and Methods

• cyclictest (from rt-tests) – measures latency of periodic timer wakeups under various loads.
• perf sched latency – records schedule events and computes average/worst‑case latency.
• /proc/sched_debug – provides per‑runqueue statistics (nr_running, avg_idle, etc.).
• Pressure Stall Information (PSI) – CPU PSI metric (some/full) correlates with scheduler stall time.

Example: Custom SCHED_FIFO Latency Measurement

/* sched_latency.c – simple SCHED_FIFO latency measurement */
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sched.h>
#include <time.h>
#include <signal.h>

static volatile sig_atomic_t stop = 0;
static void handler(int sig) { stop = 1; }

int main(int argc, char *argv[]) {
    struct sched_param param = { .sched_priority = 80 };
    if (sched_setscheduler(0, SCHED_FIFO, &param) == -1) {
        perror("sched_setscheduler");
        return 1;
    }
    signal(SIGINT, handler);

    struct timespec wt = { .tv_sec = 0, .tv_nsec = 200000 }; // 200 µs period
    struct timespec now, last;
    clock_gettime(CLOCK_MONOTONIC, &last);
    uint64_t max_latency = 0;
    uint64_t total = 0;
    unsigned long samples = 0;

    while (!stop) {
        clock_gettime(CLOCK_MONOTONIC, &now);
        uint64_t delta = (now.tv_sec - last.tv_sec) * 1000000000ULL +
                         (now.tv_nsec - last.tv_nsec);
        if (delta > wt.tv_nsec) {
            uint64_t latency = delta - wt.tv_nsec;
            if (latency > max_latency) max_latency = latency;
            total += latency;
            samples++;
        }
        last = now;
        nanosleep(&wt, NULL);
    }

    printf("Samples: %lu\n", samples);
    printf("Max latency: %llu ns\n", (unsigned long long)max_latency);
    printf("Avg latency: %llu ns\n",
           samples ? (unsigned long long)(total / samples) : 0ULL);
    return 0;
}

Compile and run:

gcc -O2 -o sched_latency sched_latency.c -lrt
sudo ./sched_latency   # requires CAP_SYS_NICE for SCHED_FIFO

The program reports worst‑case and average wakeup jitter observed while the system is under load (e.g., while a packet generator runs).

CLI Example: Using mpstat

mpstat (part of sysstat) reports per‑CPU utilization, including %irq and %soft fields that indirectly reflect time spent handling interrupts and softirqs—high values often correlate with scheduler latency spikes.

# Report every 1 second, show %irq and %soft
mpstat -P ALL 1 10

• %irq – time spent servicing hard interrupts.
• %soft – time spent in softirq context (e.g., NET_RX, NET_TX).

If %soft climbs while the runqueue length (proc/interrupts or vmstat) stays high, the scheduler is likely delaying softirq execution.

Understanding Cgroup Pressure

Types of Cgroup Pressure

Linux cgroup v2 exposes Pressure Stall Information (PSI) for three resources:

• CPU – some: fraction of time tasks are delayed because they are ready to run but cannot get CPU; full: fraction of time all tasks in the cgroup are stalled.
• Memory – some: time spent waiting for memory reclaim; full: time the cgroup is completely out of reclaimable memory.
• IO – some: time waiting for IO; full: time the cgroup is completely blocked on IO.

These metrics are available under /sys/fs/cgroup/<path>/cpu.pressure, memory.pressure, and io.pressure.

Measuring Cgroup Pressure

The psi command (from linux-tools-common) provides a convenient live view:

# Monitor CPU pressure for a specific cgroup every second
psi -p cpu -c /mycgroup 1

Alternatively, read the file directly:

cat /sys/fs/cgroup/mycgroup/cpu.pressure
# Example output:
# some avg10=0.25 avg60=0.18 avg300=0.12 total=12345678
# full avg10=0.02 avg60=0.01 avg300=0.00 total=987654

• some – percentage of time at least one task in the cgroup was stalled.
• full – percentage of time all tasks were stalled.

Example: Running Tasks with Cgroup Pressure

# Create a cgroup with a CPU quota of 50% (0.5 of a CPU)
sudo cgcreate -c cpu:/mycgroup
sudo sh -c "echo 50000 > /sys/fs/cgroup/mycgroup/cpu.max"   # 50000 µs per 100000 µs period

# Run a CPU‑bound task inside the cgroup
cgexec -g cpu:mycgroup stress --cpu 4 --timeout 60s

While stress runs, observe rising CPU pressure:

psi -p cpu -c /mycgroup 1

The some metric increases, reflecting that tasks are regularly delayed due to the quota.

IRQ Affinity and System Performance

Understanding IRQ Affinity

Each hardware interrupt has an associated IRQ descriptor under /proc/irq/<IRQ_NUMBER>/. The file smp_affinity (or smp_affinity_list in newer kernels) contains a hexadecimal mask (or CPU list) indicating which CPUs may service that interrupt.

To view:

cat /proc/irq/$(grep eth0 /proc/interrupts | awk -F: '{print $1}')/smp_affinity

To set affinity to CPU 2 only (zero‑based indexing):

echo 4 > /proc/irq/<IRQ>/smp_affinity   # 4 = binary 0100 → CPU 2

Or using the list format:

echo 2 > /proc/irq/<IRQ>/smp_affinity_list

Measuring IRQ Affinity

irqbalance is a daemon that periodically rebalances IRQs. Its status can be queried:

systemctl status irqbalance
irqbalance --debug   # one‑shot, shows current affinity and load

Sample output:

IRQ 42 (eth0-Rx) affinity: 2-3  load: 0.12

If you disable irqbalance and manually pin IRQs, you can observe the effect on packet processing latency.

Example: Pinning Softirq Daemon or Userspace Consumer

taskset sets CPU affinity for a process, not an IRQ. To affect IRQ handling you must write to the smp_affinity file directly, as shown above. However, you can pin the softirq daemon (ksoftirqd/<cpu>) to a specific CPU using taskset to study the impact:

# Pin ksoftirqd/0 to CPU 0 only (already default, but demonstrates)
sudo taskset -p 0x1 $(pgrep ksoftirqd/0)

More useful is to bind a userspace packet consumer (e.g., iperf3 -s) to the same CPU that handles the NIC IRQ, thereby testing locality:

# Assume eth0 uses IRQ 57, bound to CPU 2
sudo sh -c "echo 4 > /proc/irq/57/smp_affinity"
taskset -c 2 iperf3 -s -B 0.0.0.0   # bind server to CPU 2

Troubleshooting Scheduler Latency, Cgroup Pressure, and IRQ Affinity

Identifying Bottlenecks

1. Confirm packet loss – check ifconfig eth0 for dropped or look at /proc/net/softnet_stat (third column = dropped packets).
2. Check softirq backlog – /proc/net/softnet_stat second column shows time squeeze (when the softirq handler had to reschedule itself). Rising values indicate the CPU cannot keep up.
3. Measure scheduler latency – run cyclictest under load; watch for max latency > 100 µs (threshold depends on application).
4. Inspect cgroup pressure – read cpu.pressure; if some > 10 % under load, the cgroup is throttling.
5. Verify IRQ affinity – ensure the IRQ handling CPU is not the same as a heavily throttled cgroup CPU, unless intentional.

System Monitoring Tools for Troubleshooting

• pidstat -w – shows context switches per process; a spike indicates scheduler pressure.
• perf top – can highlight time spent in __do_softirq or scheduler_tick.
• bpftrace – one‑liners to trace wakeup latency:

# Trace the delay between a task becoming runnable and actually running
sudo bpftrace -e '
tracepoint:sched:sched_wakeup {
    @start[args->pid] = nsecs;
}
tracepoint:sched:sched_switch {
    /@start[args->prev_pid]/
    {
        @latency[args->prev_pid] = hist(nsecs - @start[args->prev_pid]);
        delete(@start[args->prev_pid]);
    }
}'

This histogram reveals scheduler latency distribution under load.

End of document.