Which drops were qdisc and which were memory

# Introduction to bpftrace and Stack-Filtered Drop Traces
bpftrace is a high-level tracing front-end for Linux eBPF that lets you attach probes to kernel functions, tracepoints, and USDT markers with a syntax similar to awk/dtrace. It compiles a short program into BPF bytecode, loads it via the bpf() syscall, and runs it in-kernel with negligible overhead when the probe frequency is low.

## Benefits of Using Stack-Filtered Drop Traces
When several counters rise simultaneously, it is hard to tell whether packet loss originates from the qdisc layer, memory-cgroup reclaim, or skb allocation failures. A stack-filtered bpftrace trace captures the full kernel stack at the point of each drop event, allowing you to:
* Attribute each drop to its exact call path.
* Correlate spikes in a specific stack with a workload or tenant namespace.
* Avoid reliance on aggregated counters that can mask the underlying cause.

## Understanding Qdisc Drops, Memcg Pressure, and Skb Allocation Failure
### Qdisc Drops: Causes and Implications
A qdisc drop occurs when the kernel’s queuing discipline cannot enqueue a packet because the qdisc’s limit is exceeded, the underlying device driver reports NETDEV_TX_BUSY, or rate-limiting shaping drops packets that exceed the configured rate.

### Memcg Pressure: Effects on System Performance
Memory cgroups (memcg) enforce limits via the memory.max and memory.low files. When a cgroup exceeds its limit, the kernel triggers reclaim, leading to indirect packet drops.

### Skb Allocation Failure: Root Causes and Consequences
`struct sk_buff` allocation can fail due to memcg pressure, fragmentation of the slab cache, or exceeding max_order for page-allocator fallback.

## Setting Up bpftrace for Drop Trace Analysis
### Installing and Configuring bpftrace
To install bpftrace, run the following command:
```bash
sudo apt-get update
sudo apt-get install -y bpftrace linux-headers-$(uname -r)

Verify the installation by running bpftrace -v.

Writing Custom bpftrace Scripts for Drop Trace Analysis

A drop trace script typically attaches to the kernel function that performs the drop, captures a user-defined stack depth, and aggregates by stack hash.

Using Stack-Filtered bpftrace Drop Traces for Incident Analysis

Identifying Noisy Multi-Tenant Incidents

In a noisy incident, you will see rising values in /proc/net/dev, increasing memory.pressure_level events, and elevated slab_alloc failures.

Separating Qdisc Drops from Memcg Pressure and Skb Allocation Failure

By attaching separate probes to each drop source and printing the kernel stack, you can produce distinct stack-trace histograms.

Example bpftrace Scripts for Drop Trace Analysis

Qdisc Drop Trace

tracepoint:net:netif_receive_skb
{
    @pkts[comm] = count();
}
kprobe:__qdisc_drop
{
    @drops[stack] = count();
}

Memcg Pressure Trace

tracepoint:memcg:memory_pressure_level
{
    if (args->level >= MEMCG_PRESSURE_MEDIUM) {
        @pressure[stack, args->level] = count();
    }
}

Skb Allocation Failure Trace

kprobe:__alloc_skb
{
    if (retval == 0) {
        @alloc_fail[stack] = count();
    }
}

Troubleshooting Common Issues with bpftrace Drop Traces

Resolving Permission and Installation Issues

• Missing kernel headers: install linux-headers-$(uname -r).
• Insufficient privileges: run with sudo or grant CAP_SYS_ADMIN via setcap cap_sys_admin+ep $(which bpftrace).

Code Examples and CLI Usage

Example bpftrace Script for Qdisc Drop Analysis

tracepoint:net:netif_receive_skb {
    @pkts[comm] = count();
}
kprobe:__qdisc_drop {
    @drops[stack] = count();
}

Execute with sudo bpftrace qdisc_drop_full.bt -t 20s.

Example bpftrace Script for Memcg Pressure Analysis

tracepoint:memcg:memory_pressure_level {
    if (args->level >= MEMCG_PRESSURE_MEDIUM) {
        @pressure[stack, args->level] = count();
    }
}

Execute with sudo bpftrace memcg_pressure_full.bt -t 20s.

Example bpftrace Script for Skb Allocation Failure Analysis

kprobe:__alloc_skb {
    if (retval == 0) {
        @alloc_fail[stack] = count();
    }
}

Execute with sudo bpftrace skb_alloc_fail_full.bt -t 20s.

Scaling Limitations and Performance Considerations

Scaling bpftrace for Large-Scale Deployments

• Per-CPU overhead: mitigate by attaching to a tracepoint or using bpftrace’s built-in sampling.
• Map size: limit stack depth and prune low-count entries periodically.

Optimizing bpftrace Performance for Low-Overhead Drop Trace Analysis

• Prefer tracepoints over kprobes.
• Use kretprobe only when necessary.
• Enable BPF JIT.