The healthy namespace that fooled the investigation

Introduction to Network Troubleshooting

Importance of Accurate Diagnosis

Accurate diagnosis is the foundation of reliable network operations. When a service experiences latency, packet loss, or connectivity failure, the operator must pinpoint the exact layer and mechanism responsible. Misdiagnosis leads to wasted effort, unnecessary configuration changes, and prolonged outages.

Challenges with Traditional Tools

Traditional packet capture tools such as tcpdump operate at a single vantage point—usually a host interface or a virtual switch port. In environments with network namespaces, containers, or virtual machines, the packet may traverse multiple namespaces before reaching the capture point. A clean trace taken in the wrong namespace shows no packets, leading to the false assumption that the traffic never left the source. This blind spot delays root‑cause analysis and encourages reliance on intuition rather than evidence.

Understanding the Problem: Clean Trace in Wrong Namespace

Namespace Isolation and Its Implications

Linux network namespaces isolate the network stack: each namespace has its own set of interfaces, routing tables, and netfilter rules. A process inside a namespace can only see traffic that belongs to that namespace. Tools that rely on raw socket captures (e.g., tcpdump) inherit the namespace of the process that invokes them unless explicitly redirected with nsenter or similar.

When a containerized application fails to communicate with an external service, a common first step is to run tcpdump on the host’s default namespace. If the traffic is actually flowing inside the container’s namespace, the host capture will be empty, producing a “clean trace.” The operator may then incorrectly conclude that the application is not sending packets, when in fact the packets are being dropped or misrouted within the isolated stack.

Case Study: Delayed Diagnosis Due to Incorrect Namespace

A three‑tier web application runs in a Kubernetes cluster. The frontend pod cannot reach the database pod. An operator runs:

tcpdump -i eth0 -nn port 5432

on a worker node and observes no packets. Assuming the frontend is silent, the operator checks application logs, finds no errors, and restarts the pod—without effect. After 45 minutes, a colleague suggests checking the pod’s namespace:

PID=$(crictl pods --name frontend -q)
NSENTER_PID=$(crictl inspectp $PID --info | grep -oP '(?<=pid": )\d+')
nsenter -n -t $NSENTER_PID tcpdump -i any -nn port 5432

Inside the pod’s namespace, the capture shows SYN packets leaving the container but no SYN‑ACK returning. The real issue is a network policy that drops the return traffic. The initial clean trace in the host namespace delayed detection of the policy‑induced drop by nearly an hour.

Introduction to bpftrace

What is bpftrace?

bpftrace is a high‑level tracing language built on Linux eBPF (extended Berkeley Packet Filter). It allows users to attach probes to kernel tracepoints, kprobes, uprobes, and USDT (user‑space) probes with a concise C‑like syntax. Programs are compiled to eBPF bytecode, loaded into the kernel, and executed with minimal overhead.

Advantages of Using bpftrace for Network Troubleshooting

Namespace‑aware filtering: bpftrace can evaluate built‑in variables such as pid, uid, cgroup, and netns to restrict tracing to a specific network namespace without leaving the host.
Arbitrary probe points: Unlike tcpdump, which only sees packets that reach a net device, bpftrace can probe functions deep in the stack (e.g., tcp_v4_rcv, ip_rcv, __netif_receive_skb) to observe where packets are processed, altered, or discarded.
Low overhead: eBPF programs run in‑kernel with a fixed‑size per‑CPU map; the impact is typically sub‑microsecond per event, suitable for production.
Custom aggregation: Users can aggregate drops by source/destination address, port, or drop reason, producing concise summaries instead of raw packet dumps.

Using bpftrace for Host and Container Socket Analysis

Installing and Setting Up bpftrace

On a recent Ubuntu/Debian host with kernel ≥ 4.18:

sudo apt-get update
sudo apt-get install -y bpftrace

Verify that the kernel has eBPF tracing enabled:

grep CONFIG_BPF_SYSCALL /boot/config-$(uname -r)   # should show =y
grep CONFIG_DEBUG_INFO_BTF /boot/config-$(uname -r) # optional for richer symbols

If running inside a container, ensure the container is privileged or has SYS_ADMIN and BPF capabilities:

docker run --rm -it --cap-add=SYS_ADMIN --cap-add=BPF \
    --security-opt apparmor=unconfined \
    ubuntu:22.04 bpftrace -v

Basic bpftrace Commands for Socket Analysis

A simple one‑liner to count sendto syscalls per process:

bpftrace -e 'tracepoint:syscalls:sys_enter_sendto { @[comm, pid] = count(); }'

To trace TCP state changes inside a specific namespace (identified by its inode):

# Find the netns inode of the target container
NSINODE=$(ls -l /proc/$(crictl inspectp <PID> --info | grep -oP '(?<=pid": )\d+')/ns/net | awk -F'[ ->]' '{print $3}')
bpftrace -v NSINODE=$NSINODE -e '
tracepoint:tcp:tcp_state_change /
    args->net == netns_to_inode(NSINODE)
/ {
    printf("%s %d -> %s\n", comm, pid, args->newstate);
}'

Example: Tracing Socket Drops with bpftrace

To observe packets dropped by the kernel’s netdev drop tracepoint, filtered to a container’s namespace:

# Determine the namespace inode first
NSINODE=$(ls -l /proc/$(crictl inspectp <PID> --info | grep -oP '(?<=pid": )\d+')/ns/net | awk -F'[ ->]' '{print $3}')
bpftrace -v NSINODE=$NSINODE -e '
tracepoint:net:net_dev_drop /
    args->net == netns_to_inode(NSINODE)
/ {
    @[args->dev_name, ntohs(args->protocol)] = count();
}
END {
    printf("Drops per device/protocol:\\n");
    print(@);
}'

The output aggregates drops by device name and Ethernet type, revealing whether loss occurs at ingress (rx) or egress (tx) and which protocol (IPv4, IPv6, ARP) is affected.

Advanced bpftrace Techniques for Network Troubleshooting

Using bpftrace to Trace TCP Connections

Tracing the lifecycle of a TCP connection from SYN to ESTABLISHED:

NSINODE=$(ls -l /proc/$(crictl inspectp <PID> --info | grep -oP '(?<=pid": )\d+')/ns/net | awk -F'[ ->]' '{print $3}')
bpftrace -v NSINODE=$NSINODE -e '
tracepoint:tcp:tcp_connect /
    args->net == netns_to_inode(NSINODE)
/ {
    @[args->saddr, args->daddr, args->dport] = join(":", args->saddr, args->daddr, args->dport);
}
tracepoint:tcp:tcp_retransmit_synack /
    args->net == netns_to_inode(NSINODE)
/ {
    printf("SYN‑ACK retransmit for %s\\n", join(":", args->saddr, args->daddr, args->dport));
}'

Analyzing Socket Buffers and Queue Lengths

Monitoring the sk_rmem_alloc and sk_wmem_alloc fields to detect receive‑side buffer pressure:

NSINODE=$(ls -l /proc/$(crictl inspectp <PID> --info | grep -oP '(?<=pid": )\d+')/ns/net | awk -F'[ ->]' '{print $3}')
bpftrace -v NSINODE=$NSINODE -e '
kprobe:tcp_v4_rcv /
    args->net == netns_to_inode(NSINODE)
/ {
    @rx[comm] = hist(args->sk->sk_rmem_alloc);
}
kprobe:tcp_v4_sendmsg /
    args->net == netns_to_inode(NSINODE)
/ {
    @tx[comm] = hist(args->sk->sk_wmem_alloc);
}
'

Histograms expose whether a particular process is consistently starving for buffer space.

Example: Debugging TCP Connection Issues with bpftrace

A service reports “connection timed out” when reaching an external API. The following script traces SYN packets, SYN‑ACK replies, and retransmissions, then summarizes per‑destination:

NSINODE=$(ls -l /proc/$(crictl inspectp <PID> --info | grep -oP '(?<=pid": )\d+')/ns/net | awk -F'[ ->]' '{print $3}')
bpftrace -v NSINODE=$NSINODE -e '
tracepoint:tcp:tcp_connect /
    args->net == netns_to_inode(NSINODE)
/ {
    @syn[args->daddr, args->dport] = count();
}
tracepoint:tcp:tcp_retransmit_syn /
    args->net == netns_to_inode(NSINODE)
/ {
    @retrans[args->daddr, args->dport] = count