Why tcpdump in the Pod Sees Nothing

Introduction to Packet Flow

The Linux network stack processes packets from the NIC through kernel layers (data link, network, transport) before delivering them to an application. In a containerized environment, a packet arriving at the host NIC is first handled by the host kernel, then steered into the container’s network namespace where the container’s stack processes it.

Host NIC → Linux Kernel → Host Interface → Container Namespace → Container Network Stack

Example capture on the host:

tcpdump -i eth0 -n -vv -s 0 -c 1

Inside a container (using its PID):

nsenter -t <container_pid> -n tcpdump -i eth0 -n -vv -s 0 -c 1

XDP and Traffic Dropping/Redirecting

XDP Overview

eXpress Data Path (XDP) lets you run an eBPF program at the earliest point in the NIC driver, enabling high‑speed packet filtering, dropping, or redirecting.

Loading an XDP Program

# Compile your eBPF program to xdp_drop.o (returns XDP_DROP for matching packets)
sudo ip link set dev eth0 xdp obj xdp_drop.o sec xdp

To verify:

ip link show dev eth0   # look for "xdp" flag

To remove:

sudo ip link set dev eth0 xdp off

Effect on Packet Flow

• Drop – packet is discarded by the NIC driver; it never reaches the host kernel, so no further forwarding occurs.
• Redirect – packet is steered to another NIC, queue, or the generic networking stack, altering the normal path.

If XDP drops a packet before the host kernel processes it, the container never sees the packet, and any tcpdump inside the container will be silent.

Why Pod‑Side Tcpdump Stays Silent

When XDP drops or redirects traffic early, the packet never arrives in the container’s network namespace. Consequently:

• Host‑side tcpdump (still attached to eth0) can see the packet because it captures at the NIC before XDP processing.
• Container‑side tcpdump (run via nsenter) sees nothing because the packet is absent from that namespace.

Example:

# Host capture – still sees packets even if XDP drops them
tcpdump -i eth0 -n -vv -s 0 -c 1

# Container capture – silent when XDP has dropped/redirected the flow
nsenter -t <container_pid> -n tcpdump -i eth0 -n -vv -s 0 -c 1

Namespace and Interface Combinations for Trustworthy Evidence

Using the correct namespace/interface pair ensures you are observing the packet at the intended point in the stack.

Troubleshooting Tcpdump Silence

1. Check XDP state

   ip link show dev eth0   # verify xdp flag and program name

2. List loaded XDP programs (if using iproute2)

   # Shows prog ID; you can then inspect with bpftool
   ip -det link show dev eth0

3. Inspect the eBPF program (optional)

   bpftool prog show id <prog_id>
   bpftool prog dump xlated id <prog_id>

4. Validate packet flow
   • Host capture: tcpdump -i eth0 ...
   • Container capture: nsenter -t <container_pid> -n tcpdump -i eth0 ...
5. If XDP is dropping/redirecting unintentionally, adjust the eBPF logic or unload the program:

   sudo ip link set dev eth0 xdp off

Scaling Limitations and Considerations

• XDP – limited by NIC driver queue size and the complexity of the eBPF program; heavy programs can increase CPU cycles per packet.
• tcpdump – packet capture overhead grows with line rate; large buffer sizes or ring buffers (-B) may be needed to avoid drops.
• Best practices
  • Deploy lightweight XDP programs for filtering; offload complex logic to later stack layers.
  • Use multiple tcpdump instances or tools like pcap with ring buffers for high‑speed capture.
  • Monitor XDP packet counters (ip -stat link show dev eth0) and tcpdrop statistics to detect loss.

Advanced Tcpdump Techniques

Namespace‑aware capture

# Host NIC
tcpdump -i eth0 -n -vv -s 0 -c 1

# Container namespace
nsenter -t <container_pid> -n tcpdump -i eth0 -n -vv -s 0 -c 1

Capturing at specific stack points

• Pre‑XDP (NIC) – tcpdump -i eth0
• Post‑XDP, pre‑stack – use tc ingress mirroring or veth pair taps.
• Inside container – as shown above with nsenter.

Example: mirroring ingress traffic to a tap for inspection

# Create a veth pair
ip link add veth0 type veth peer name veth1
ip link set veth1 up
# Mirror ingress eth0 to veth1
tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: protocol all bpf da obj xdp_mirror.o sec xdp
# Capture on the mirror interface
tcpdump -i veth1 -n -vv -s 0 -c 1

Real‑World Examples and Case Studies

XDP‑Based DDoS Mitigation

An XDP program drops UDP packets exceeding a rate threshold per source IP, preventing malicious traffic from entering the host stack. Host‑side tcpdump sees the dropped packets; container‑side tcpdump sees only legitimate traffic.

Tcpdump‑Driven Network Troubleshooting

Intermittent packet loss in a microservice was traced to an XDP redirect that sent traffic to an unused NIC. By comparing host‑side and container‑side captures, the mis‑directed flow was identified and the XDP program corrected.

Lessons Learned

• Validate XDP programs in a test environment before production rollout.
• Always verify packet visibility at both host and container levels when troubleshooting.
• Use namespace‑specific capture (nsenter) to confirm whether a packet has reached the container stack.