Kernel version changes in drop tracing cost

Introduction to Benchmarking skb Drop Reason Tracing

The Linux socket buffer (skb) carries a drop_reason field that is set when a packet is discarded by the networking stack. Tracing this field lets operators see why packets are dropped (congestion, ACL, RPF failure, LRO overflow, etc.) without touching the data path. The mechanism relies on eBPF programs attached to kernel probes that fire when an skb is freed or consumed, read the drop reason via bpf_skb_get_drop_reason(), and record it in a userspace‑visible map.

Production observability must stay low‑overhead. Adding a tracing probe consumes CPU cycles, memory bandwidth, and can cause contention on shared resources (per‑CPU maps, ring buffers). Benchmarking across kernel releases quantifies that overhead so teams can:

• Verify the tracing cost fits a latency budget (e.g., < 2 µs per packet at 10 Mpps).
• Spot regressions caused by helper changes or symbol renames.
• Pick the probe type (kprobe, tracepoint, fentry/fexit) that gives the best visibility‑overhead trade‑off for a given kernel version.

Kernel Versions and Probe Choice

Kernel Overview

All three ship with CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, and CONFIG_DEBUG_INFO_BPF=y in typical distro configs, enabling libbpf‑based tracing.

Probe Options

Choosing a Probe for Benchmarking

• Tracepoint – preferred when the tracepoint exists and is stable across kernels (it does for kfree_skb). It eliminates symbol‑resolution variance and yields deterministic overhead.
• Fentry/Fexit – chosen when tracepoint granularity is insufficient (e.g., you need to capture arguments before they are overwritten) and the kernel version supports BPF trampoline (≥ 5.5).
• Kprobe – serves as a fallback on kernels lacking fentry support or when you need to probe a function that lacks a tracepoint (rare for skb free paths).

For the benchmark we compare:

1. Tracepoint on kfree_skb.
2. Fentry on __kfree_skb (when available).
3. Kprobe on __kfree_skb (as a baseline).

Helper Availability and Its Impact

Core Helpers

• bpf_skb_get_drop_reason(struct sk_buff *skb) – returns the enum stored in skb->drop_reason.
• bpf_skb_set_drop_reason(struct sk_buff *skb, u32 reason) – mainly for testing; not needed for passive observation.

Frequently used alongside:

• bpf_probe_read_kernel() – safe field reads when BPF_CORE_READ isn’t usable.
• bpf_get_smp_processor_id() – per‑CPU map indexing.
• bpf_ringbuf_reserve() / bpf_ringbuf_submit() – low‑latency event output.
• bpf_map_lookup_elem() / bpf_map_update_elem() – histogram aggregation.

Helper Availability Across Kernels

Overall, the helper set relevant to drop‑reason tracing is stable from 5.15 → 6.6; performance differences arise from probe choice and kernel internals (inlining, layout changes), not helper availability.

Symbol Stability and Its Effect

What Makes a Symbol Stable?

A symbol is stable for tracing when its name persists across builds and its address can be reliably resolved via /proc/kallsyms or BPF CO‑RE. Instability affects:

• Kprobe/Fentry attachment – requires the exact symbol name; a rename forces re‑attachment or failure.
• Tracepoint usage – immune to symbol changes because tracepoints are identified by a static string in the tracepoint definition.
• BPF CO‑RE – relies on struct field offsets; if a struct layout changes, regenerate vmlinux.h, but the probe itself does not break.

Symbol Stability per Kernel

• 5.15 – __kfree_skb is not static inline; it appears as an exported symbol in kallsyms. Kprobe/fentry on __kfree_skb works reliably. The tracepoint kfree_skb (in trace/events/skb.h) has been unchanged since 4.14. No known renames in the skb free path.

• 6.1 – __kfree_skb received __always_inline under certain configs (e.g., CONFIG_OPTIMIZE_INLINING=y). When inlined, the separate symbol may disappear from kallsyms, breaking a raw kprobe. The tracepoint kfree_skb remains unaffected because it sits at the call site. Fentry/Fexit probes, which use the BPF trampoline, can still attach to the inlined function via the compiler‑generated call site, making them more resilient than raw kprobes.

• 6.6 – Inlining of __kfree_skb is now the default in most distro builds. The tracepoint kfree_skb is still present and stable. Relying on a kprobe is risky unless the kernel is built with CONFIG_DEBUG_INFO_BPF=y and CONFIG_KPROBES_SANITIZE=y (which forces the function out‑of‑line for debugging). Fentry/Fexit remain the recommended dynamic probe choice for 6.6 when a tracepoint does not provide the needed granularity.

Benchmarking Methodology and Tools

Tools

• Traffic Generator – pktgen (built‑in) or trex for controllable UDP streams (1–10 Mpps, 64‑byte packets).
• Latency Measurement – cyclictest (baseline) and hwlatdetector to isolate jitter; per‑packet latency captured via SO_TIMESTAMPING (SCM_TSTAMP_*).
• CPU Overhead – perf stat -e cycles,instructions,cache-references,cache-misses applied to the tracing process; also pidstat -p <pid> 1 for context‑switch count.
• eBPF Program – written with libbpf (v0.8+), compiled with clang -target bpf -O2 -g. The program attaches to the selected probe, reads the drop reason via bpf_skb_get_drop_reason(), increments a per‑CPU histogram (struct { __u64 slots[8]; }), and optionally submits events to a ringbuf for userspace consumption.
• Baseline – identical traffic flow with no eBPF program loaded (or a nop program that merely returns 0).

Procedure

1. Prepare the testbed – two servers linked by a 25 GbE NIC (e.g., Mellanox ConnectX‑6) with flow control disabled, RSS enabled, and gro=off gso=off tso=off to prevent offload‑induced masking.
2. Kernel configuration – ensure CONFIG_BPF_SYSCALL=y, CONFIG_BPF_JIT=y, CONFIG_DEBUG_INFO_BPF=y, CONFIG_TRACING=y, CONFIG_BPF_TRACING=y, CONFIG_KPROBES=y, CONFIG_FTRACE=y. Verify the tracepoint exists:

   grep kfree_skb /sys/kernel/debug/tracing/available_events

3. Build the eBPF program – three variants (tracepoint, fentry, kprobe) using the same source; only the SEC() annotation changes.
4. Run the benchmark – start traffic, attach the program, collect latency and CPU metrics, then detach and repeat for each probe/kernel combination.
5. Analyze – compare per‑packet overhead, jitter, and CPU usage against the baseline to determine which probe offers the best trade‑off for each kernel version.

End of document.