The namespace was deleted but the traffic survived

Introduction to Namespace Pinning

Overview of Namespace Lifecycle

A Linux namespace is created when a process calls clone(2) with CLONE_NEW* flags or unshare(2). The kernel allocates a namespace object and increments its reference count. The namespace persists until the count drops to zero: every process that has the namespace open (via a file descriptor in /proc/<pid>/ns/) or a bind mount that references it holds a reference. When the last reference is released, the kernel destroys the namespace and all associated resources (e.g., network devices, mounts).
Operators often assume that stopping the “main” process of a container or releasing a network interface removes the namespace, but hidden references keep it alive.

Understanding File Descriptors, Bind Mounts, and Long‑Lived Processes

Three common sources of stray references:

• File descriptors – an open /proc/<pid>/ns/<type> fd or a socket bound to a network namespace.
• Bind mounts – a mount namespace that bind‑mounts /proc/<pid>/ns/<type> or a directory inside the namespace (e.g., /var/run/netns).
• Long‑lived processes – a daemon, side‑car, or debugging tool that entered the namespace via setns(2) and never exited or closed its fd.

Any of these can pin the namespace after the operator believes it is unused, causing datapath leaks, address‑conflict errors, or inability to reuse a VLAN/VNI.

File Descriptors and Namespace Pinning

How File Descriptors Keep Namespaces Alive

When a process opens /proc/<pid>/ns/net (or any ns type) the kernel increments the namespace’s reference count. The fd inherits across fork and exec unless FD_CLOEXEC is set. A process that calls setns(fd, 0) also holds a reference until it either exits or closes the fd. If the fd is leaked (e.g., forgotten in a library, left open by a side‑car, or inherited by a shell), the namespace cannot be torn down.

Identifying File Descriptor‑Related Issues

Look for processes that have open ns fds but are not the expected owner. lsof can list those fds; ls -l /proc/<pid>/fd shows the symlink targets.

Example: Using lsof to Detect File Descriptor Leaks

# Suppose we suspect the netns for container id abc123 is stuck
# Find the pid of the container's init (e.g., via docker inspect)
CONTAINER_PID=$(docker inspect -f '{{.State.Pid}}' abc123)
# List all processes with an open fd to that netns
lsof -p $(pgrep -d, -f ".*") | grep "/proc/${CONTAINER_PID}/ns/net"

Sample output (annotated):

sshd      5678 root    3u      REG                0,23        0  12345 /proc/1234/ns/net

Notice: fd 3 of sshd (pid 5678) points to /proc/1234/ns/net. The sshd process inherited the fd from its parent and never closed it, keeping the container’s network namespace alive even after the container’s main process exited.

Bind Mounts and Namespace Persistence

The Role of Bind Mounts in Namespace Pinning

A bind mount does not copy data; it creates another reference to the same underlying mount point or namespace file. If a bind mount references /proc/<pid>/ns/<type> (commonly done by tools like ip netns that store namespaces under /var/run/netns), the kernel counts that mount as a reference. Unmounting the bind mount drops the reference; leaving it mounted pins the namespace.

Troubleshooting Bind Mount‑Related Issues

Check /proc/mounts or mount for entries that reference ns paths. Look for stale entries under /var/run/netns or custom debug directories.

Code Example: Using mount and umount to Manage Bind Mounts

# List bind mounts that reference a network namespace
mount | grep "/proc/.*/ns/net"
# Sample output:
# /var/run/netns/abc123 on /proc/3456/ns/net type none (rw,bind)

Notice: the bind mount /var/run/netns/abc123 -> /proc/3456/ns/net holds a reference.

To release:

umount /var/run/netns/abc123
# Verify it's gone
mount | grep abc123 || echo "no bind mount"

If the umount fails with “target is busy,” another process still has an fd open; repeat the fd check.

Long‑Lived Processes and Namespace Pinning

How Long‑Lived Processes Affect Namespace Lifecycle

A process that enters a namespace via setns(2) and then performs long‑running work (e.g., a monitoring agent, a side‑car proxy, or a debugging shell) retains a reference for its entire lifetime. If the process is not designed to exit when the namespace is no longer needed, it becomes a pinning agent.

Identifying and Managing Long‑Lived Processes

Use ps to find processes that have a namespace fd open or that have a non‑default NSpid/NSpgid in /proc/<pid>/status. The ip netns identify helper can also map a pid to its netns.

CLI Example: Using ps and kill to Manage Processes

# Find all processes that have a reference to netns of pid 3456
for p in $(ps -e -o pid); do
    if ls -l /proc/$p/fd/ 2>/dev/null | grep -q "/proc/3456/ns/net"; then
        echo "pid $p holds ns fd"
    fi
done

Sample output (annotated):

pid 7890 holds ns fd
pid 7892 holds ns fd

Notice: two processes (likely a monitoring agent and a debug shell) still have the ns fd open.

To remove the pin:

kill -TERM 7890 7892   # or kill -9 if they ignore TERM
# Re‑check
for p in $(ps -e -o pid); do
    ls -l /proc/$p/fd/ 2>/dev/null | grep -q "/proc/3456/ns/net" && echo "still held by $p"
done
# No output → references released

After the last reference is dropped, the netns can be removed (ip netns delete) or will auto‑destruct when the last process exits.

Troubleshooting Namespace Pinning Issues

Using Raw Artifacts to Analyze Namespace State

Raw artifacts expose the kernel’s reference‑count view. The most direct is /proc/sys/kernel/ns_last_pid (shows the last allocated ns identifier) and per‑ns files under /proc/<pid>/ns/. Comparing the inode numbers of those symlinks with what mount and lsof report reveals leaks.

Example: Analyzing proc and sys Filesystems for Clues

# Get the inode of the network namespace we expect to be free
IPNETNS_INODE=$(ls -l /proc/3456/ns/net | awk -F'[ >]' '{print $5}')
echo "Expected netns inode: $IPNETNS_INODE"
# Find any processes or mounts referencing that inode
echo "Processes with fd to that inode:"
grep -r "$IPNETNS_INODE" /proc/*/fd/ 2>/dev/null | cut -d/ -f3 | sort -u
echo "Mount points referencing that inode:"
grep "$IPNETNS_INODE" /proc/mounts

Sample output (annotated):

Expected netns inode: 4026532293
Processes with fd to that inode:
5678
7890
7892
Mount points referencing that inode:
/var/run/netns/abc123 /proc/3456/ns/net none rw,bind 0 0

Notice: the inode appears in three process fd lists and one bind mount. All must be cleared.

Step‑by‑Step Guide to Identifying and Resolving Namespace Pinning Issues

1. Identify the suspect namespace – obtain its inode via ls -l /proc/<pid>/ns/<type>.
2. List all fd references – grep -r <inode> /proc/*/fd/.
3. List all bind‑mount references – grep <inode> /proc/mounts.
4. For each pid, decide if the process is legitimate; if not, stop it or close the fd (gdb -p <pid> -ex "call close(<fd>) -ex detach").
5. Umount any stale bind mounts.
6. Re‑check – no fd or mount should reference the inode.
7. Confirm namespace death – ip netns list no longer shows the name, or attempting setns returns ENOENT.

Scaling Limitations and Namespace Pinning

How Namespace Pinning Affects System Scalability

Each pinned namespace consumes kernel memory (struct nsproxy, network device objects, iproute tables). In large‑scale environments (thousands of containers or VMs), a small leak rate can exhaust the max_user_namespaces limit or cause slab allocation failures, leading to ENOMEM when creating new namespaces.

Mitigating Scaling Limitations with Proper Namespace Management

Impose strict lifecycle policies:

• Automatic cleanup – use ip netns add with a corresponding ip netns delete in the same unit file or container init script.
• Reference‑count monitoring – expose the number of ns fds via a custom metric (see later).
• Isolate debugging tools – run them in their own temporary namespace (unshare -n -- bash) so they do not inherit the production ns fd.

Example: Using systemd and cgroups to Manage Resource Allocation

A systemd service can limit the number of namespaces a unit may create via Delegate=yes and CPUQuota=/MemoryLimit= indirectly, but the direct knob is TasksMax= (limits threads, which indirectly bounds ns creation). More precise is a cgroups v2 max on the pids controller combined with a systemd ExecStartPre= that runs a cleanup script.

# /etc/systemd/system/container-netns@.service
[Unit]
Description=Network namespace for %i
After=network-local.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/ip netns add %i
ExecStop=/usr/bin/ip netns delete %i
# Ensure no lingered processes
ExecStopPost=/usr/bin/fuser -k /var/run/netns/%i || true
# Limit total number of netns instances across all services
Slice=system-netns.slice
[Install]
WantedBy=multi-user.target

Corresponding slice:

# /etc/systemd/system/system-netns.slice
[Slice]
CPUQuota=50%
MemoryMax=1G
TasksMax=5000   # upper bound on threads, thus on ns creation

Notice: the service creates and deletes the netns in lockstep with its lifecycle; the slice caps total concurrent namespaces.

Raw Artifact Analysis for Namespace Pinning

Using strace and tcpdump