Why cloned images lose tcpdump and raw sockets

Introduction to Troubleshooting Custom Node Images

When building a custom node image for a lab or production environment, the intent is to encapsulate a known set of binaries, libraries, and runtime constraints (capabilities, seccomp filters, device nodes) that match the workload’s expectations. In practice, the image may boot without error, yet packet capture tools (e.g., tcpdump, dumpcap) or routing daemons (e.g., frr, bird, quagga) exhibit silent failures: they start, but no packets are seen, or routing tables remain empty. The root cause is often a drift between the build‑time security profile (what the Dockerfile or build script assumed) and the runtime enforcement applied by the container runtime or orchestrator. This drift can be introduced by:

• Default Docker seccomp profile dropping syscalls needed for packet capture (socket, bind, setsockopt with SO_ATTACH_FILTER, etc.).
• Dropping Linux capabilities such as CAP_NET_RAW or CAP_SYS_ADMIN that raw socket access depends on.
• Device node permissions changed by the runtime (e.g., /dev/net/tun owned by root, or missing cgroup device allow rules).

Identifying the symptom first—daemon runs but produces no output—then tracing the enforcement mechanisms narrows the search space.

Identifying Key Components: Packet Capture and Routing Daemons

Packet Capture

Packet capture in containers typically relies on:

• Raw socket (AF_PACKET) access, requiring CAP_NET_RAW and, on some kernels, CAP_SYS_ADMIN for certain socket options.
• libpcap, which may invoke socket(), bind(), setsockopt(), and ioctl() on the socket descriptor.
• Access to /dev/net/tun when using TAP‑based capture (less common but present in some FRR plugins).

Routing Daemons

Routing daemons need:

• CAP_NET_ADMIN to modify interfaces, add/delete routes, and manage neighbor tables.
• CAP_NET_RAW for sending/receiving raw packets (e.g., OSPF hello, BGP TCP with MD5).
• Access to /dev/net/tun if they create virtual interfaces (e.g., FRR’s zebra with tun driver).
• Ability to bind to privileged ports (<1024) for certain protocols (requires CAP_NET_BIND_SERVICE).

If any of these are missing at runtime, the daemon will log a generic “permission denied” or fail silently, depending on how error handling is implemented.

Analyzing Capabilities and Seccomp Settings

Overview of Linux Capabilities

Linux capabilities split root privileges into distinct bits that can be independently granted or dropped. Relevant bits for our use case:

A container launched without explicit --cap-add inherits a default set defined by the runtime (Docker, containerd, cri-o). The default set excludes CAP_NET_RAW and CAP_NET_ADMIN. If the image build assumed those capabilities were present (e.g., the Dockerfile RUN setcap cap_net_raw+ep /usr/sbin/tcpdump), the binary will have the capability flag set, but the kernel will still reject the operation if the capability is not in the thread’s permitted set.

Understanding Seccomp and Its Impact on Daemon Behavior

Seccomp (secure computing mode) filters syscalls via a BPF program. Docker’s default profile (/etc/docker/default.json or built‑in) allows a broad set but denies several that are rarely needed in typical containers, including:

• bpf – used by libpcap to attach a filter program to a socket.
• setuid, setgid – sometimes invoked by daemons dropping privileges.
• ptrace – used by debugging tools, not needed for normal operation.
• clone with CLONE_NEWNET – needed if the daemon creates its own network namespace.

If a required syscall is filtered, the thread receives SIGSYS (signal 31) and, unless handled, terminates with “Illegal instruction”. Many daemons do not install a handler for SIGSYS, resulting in an abrupt exit that may be missed if logs are not captured.

Example: Using capsh to Analyze Capability Settings

capsh (from libcap2) can decode the current capability state of a process.

# Inside a running container
$ capsh --print
Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_audit_control+ep
Bounding set = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_audit_control
Ambient set =

Notice CAP_NET_RAW appears in the effective set (+ep) only if the runtime granted it. If you see cap_net_raw missing, the daemon lacks raw socket permission.

To test a specific binary:

$ capsh --print --inh="cap_net_raw,cap_net_admin" -- -c "id && /usr/sbin/tcpdump -i any -c 1 -w /dev/null 2>&1"

If the binary fails with “Operation not permitted”, the capability set is insufficient.

Example: Using seccomp Tools to Inspect Seccomp Profiles

Docker’s default seccomp profile can be viewed with:

$ docker run --rm --security-opt seccomp=unconfined alpine cat /etc/docker/default.json | jq '.syscalls[] | select(.name=="bpf")'

If the output is empty, the bpf syscall is blocked. A more direct inspection from inside a container:

$ grep -E 'bpf|socket|setsockopt' /proc/self/status   # shows Seccomp: 2 (enabled) and the filter bits
$ cat /proc/$$/status | grep Seccomp
Seccomp:        2

To see which syscalls are actually being blocked, enable audit:

# On the host (requires auditd)
$ sudo auditctl -a exit,always -S bpf -F auid>=1000 -F auid!=-1 -k bpf_calls
# Then run the container and check:
$ sudo ausearch -k bpf_calls --raw | aureport -f -i

If you see bpf denied, that explains why tcpdump cannot attach a filter.

Investigating Device Access Changes

Device Access Control: An Overview

Device nodes are governed by:

• Filesystem permissions (ls -l /dev/*).
• udev rules that can change ownership, mode, or add tags.
• cgroup v2 device controller (if used) that whitelists or blacklists access to specific major/minor numbers.
• Container runtime device allow/deny lists (--device, --device-cgroup-rule).

A common scenario: the image build expects /dev/net/tun to be crw-rw-rw- (owned by root, group netdev, mode 666) so that an unprivileged user can open it. At runtime, the container may start with a restrictive device cgroup that only allows c 10:200 rwm (tun) but the node appears as crw-rw---- (root:root 660) because the runtime did not apply the expected udev rule, causing open("/dev/net/tun", O_RDWR) to fail with EACCES.

Using udev Rules to Manage Device Access

Udev rules live in /etc/udev/rules.d/ (host) or can be injected into the image via /usr/lib/udev/rules.d/. A typical rule for tun:

KERNEL=="tun", GROUP="netdev", MODE="0660"

If the container’s /etc/udev/rules.d/ lacks this file, the device defaults to root:root 600.

Example: Troubleshooting Device Access Issues with udevadm

Inside the container:

$ udevadm info -a -n /dev/net/tun | grep '{group}' | head -1
  looking at device '/devices/virtual/net/tun':
    ATTR{group}=="0"

If group is 0 (root) instead of netdev, the rule didn’t apply.

You can also test rule application:

$ udevadm test /dev/net/tun 2>&1 | grep GROUP

CLI Example: Checking Device Permissions with ls -l /dev/

$ ls -l /dev/net/tun
crw-rw---- 1 root root 10, 200 Sep 25 12:34 /dev/net/tun

Compare with the expected crw-rw-rw- or crw-rw---- root netdev. If the group is wrong, either adjust the runtime device rule:

# Docker run example
$ docker run --rm -it \
    --device /dev/net/tun \
    --group-add $(getent group netdev | cut -d: -f3) \
    myimage

Or rebuild the image with a proper udev rule and ensure the runtime preserves it (--udev flag in podman, or --privileged as a last resort).

Troubleshooting Packet Capture Issues

Common Packet Capture Tools and Their Requirements

If any of these are missing, the tool will either exit immediately with an error or appear to hang (waiting on a socket that never receives packets).

Troubleshooting Steps for Packet Capture Daemons

1. Verify the daemon process is alive (ps -ef | grep tcpdump).
2. Check its open file descriptors (ls -l /proc/<pid>/fd/). Look for a socket of type packet.
3. Inspect capability bounding set via /proc/<pid>/status (CapBnd).
4. Look for seccomp denials in dmesg or audit logs (grep -i bpf /var/log/audit/audit.log).
5. Test raw socket creation manually with a small C program or python -c "import socket; s=socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.htons(0x0003)); print(s)".
6. If using TAP, ensure /dev/net/tun is accessible (strace -e openat tcpdump -i tun0 -c 1 2>&1 | grep openat).

Example: Using tcpdump to Verify Packet Capture Functionality

# Inside container, assume we have a veth pair linked to a namespace with traffic
$ ip link add veth0 type veth peer name veth1
$ ip link set veth0 up
$ ip link set veth1 netns 1
$ ip netns exec 1 ip link set veth1 up
$ ip netns exec 1 ip addr add 10.0.0.2/24 dev veth1
$ ip netns exec 1 ping -c 2 10.0.0.1 &
# Now capture
$ timeout 2 tcpdump -i veth0 -nn -c 5

Expected output shows ICMP echo request/reply. If you see “no packets captured” or the command exits instantly with “Operation not permitted”, the problem lies in capabilities/seccomp.

Code Example: Implementing Packet Capture with libpcap

/* simple_capture.c – compile with: gcc -lpcap simple_capture.c -o simple_capture */
#include <pcap.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
    char errbuf[PCAP_ERRBUF_SIZE];
    pcap_t *handle = pcap_open_live(argv[1], BUFSIZ, 1, 1000, errbuf);
    if (!handle) {
        fprintf(stderr, "Couldn't open device %s: %s\n", argv[1], errbuf);
        return 2;
    }
    struct pcap_pkthdr header;
    const u_char *packet;
    int res = pcap_next_ex(handle, &header, &packet);
    if (res == 1) {
        printf("Got a packet of length %d\n", header.len);
    } else if (res == -1) {
        fprintf(stderr, "Error while reading: %s\n", pcap_geterr(handle));
    }
    pcap_close(handle);
    return 0;
}

Run it inside the container to confirm whether the underlying capture mechanism works. If it fails, revisit the capability and seccomp checks above.