When Local Success Hides a Shared-Runner Failure

Introduction

A Containerlab topology that spins up FRR‑based routers connected via Linux bridges works reliably on a developer laptop but fails intermittently on shared CI runners (GitLab‑shared‑shell or GitHub‑Actions self‑hosted). Nodes may never obtain an IP address, BGP sessions stay in Idle, and clab destroy leaves orphaned bridges and network namespaces. The laptop’s host state is reset between runs (or manually cleaned), while the CI host accumulates residue from previous jobs, eventually hitting hidden resource limits.

These differences create a boundary condition: the lab works until a hidden resource (bridge ifindex, network namespace, or cgroup limit) is exhausted, at which point node creation fails silently.

Identifying the Root Cause

Stale Bridges

After a failed run on the CI host:

$ ip -br link show type bridge
UP             br-eth0        ...
UP             br-eth1        ...
DOWN           br-eth2        ...
DOWN           br-eth3        ...
...

Each br-ethX is a point‑to‑point link created by Containerlab. A successful clab destroy should bring them down and delete them. On the CI host many remain in DOWN state, consuming an ifindex each.

Leaked Namespaces

Similarly, network namespaces linger:

$ ip netns list
clab‑lab1‑r1
clab‑lab1‑r2
clab‑lab2‑r1
clab‑lab2‑r2
...

Each namespace should be removed when its container stops. The CI host shows dozens of orphaned namespaces from previous jobs, each counting toward the user namespace limit.

Host Resource Limits

Key limits that were hit:

• /proc/sys/user/max_user_namespaces – CI host set to 5000; after ~4800 namespaces the kernel returns EAGAIN on unshare(CLONE_NEWNET).
• Ifindex space – effectively limited to ~2^20; after ~1.2M interfaces the kernel refuses new device registration (RTNETLINK answers: Operation not permitted).
• cgroup v2 pids.max – set to 2048; a lab that creates 30 FRR containers each spawning 20 threads can exceed this, causing fork() failures.
• /proc/sys/fs/file-max – low on the shared host; each bridge and namespace consumes a file descriptor for netlink sockets, leading to Too many open files.

The laptop never approached these thresholds because it was either rebooted between tests or the user manually cleaned up.

Troubleshooting Techniques

Checking Bridge Configuration

# List all bridges and their state
ip -br link show type bridge

# Show detailed info for a suspect bridge
ip link show dev br-eth42

# Count bridge interfaces
grep -c '^br-' /proc/net/dev

Inspecting Namespace Usage

# Count existing network namespaces
ip netns list | wc -l

# Show bind‑mount status for each namespace
for ns in $(ip netns list); do
    echo "Namespace $ns:"
    ls -l /run/netns/$ns 2>/dev/null || echo "  (no bind mount)"
done

# Detect orphaned namespaces (no bind mount)
for ns in $(ip netns list); do
    if ! mountpoint -q /run/netns/$ns; then
        echo "Orphaned: $ns"
    fi
done

Monitoring Host Resource Utilization

# User namespace limit and usage
cat /proc/sys/user/max_user_namespaces
cat /proc/sys/kernel/ns_last_pid 2>/dev/null || echo 'n/a'

# cgroup v2 stats (if present)
if [ -d /sys/fs/cgroup ]; then
    echo "pids.max:   $(cat /sys/fs/cgroup/pids.max)"
    echo "pids.cur:   $(cat /sys/fs/cgroup/pids.current)"
    echo "memory.max: $(cat /sys/fs/cgroup/memory.max)"
    echo "memory.cur: $(cat /sys/fs/cgroup/memory.current)"
fi

# File descriptor usage for the current process
ls -l /proc/$$/fd | wc -l

CLI Tools for Diagnosis

• brctl show – legacy bridge tool, useful if ip output is truncated.
• nsenter --net=/var/run/netns/<name> bash – enter a namespace to verify connectivity.
• cgroupstat (from libcgroup-tools) – hierarchical view of cgroup usage.
• systemd-cgtop – real‑time view of cgroup pressure.
• perf trace -t sys_clone,sys_unshare – see if unshare calls are failing with EAGAIN.

Code Examples for Troubleshooting

Bridge Cleanup Script

#!/usr/bin/env bash
# cleanup-bridges.sh – remove all Containerlab‑created bridges
set -euo pipefail

PREFIX="br-eth"
for br in $(ip -br link show type bridge | awk -v p="$PREFIX" '$1 ~ "^"p {print $1}'); do
    echo "Removing bridge $br"
    ip link set dev "$br" down
    ip link delete dev "$br" type bridge
done

Run as a pre‑job step: ./cleanup-bridges.sh || true.

Namespace Cleanup Script

#!/usr/bin/env bash
# cleanup-namespaces.sh – delete all Containerlab network namespaces
set -euo pipefail

PREFIX="clab-"
for ns in $(ip netns list | grep "^$PREFIX"); do
    echo "Deleting namespace $ns"
    ip netns delete "$ns"
done

Invoke before each clab deploy.

Resource Monitoring Snapshot

#!/usr/bin/env bash
# monitor-limits.sh – snapshot key limits and usage
set -euo pipefail

echo "=== User namespaces ==="
echo "limit: $(cat /proc/sys/user/max_user_namespaces)"
echo "used:  $(cat /proc/sys/kernel/ns_last_pid 2>/dev/null || echo 'n/a')"

echo "=== cgroup v2 (if present) ==="
if [ -d /sys/fs/cgroup ]; then
    echo "pids.max:   $(cat /sys/fs/cgroup/pids.max)"
    echo "pids.cur:   $(cat /sys/fs/cgroup/pids.current)"
    echo "memory.max: $(cat /sys/fs/cgroup/memory.max)"
    echo "memory.cur: $(cat /sys/fs/cgroup/memory.current)"
fi

echo "=== File descriptors (process $$) ==="
ls -l /proc/$$/fd | wc -l

Add this as a debug step in CI to capture trending values.

Scaling Limitations and Boundary Conditions

Understanding Resource Constraints

• Network namespace limit – governed by max_user_namespaces. Each unshare(CLONE_NEWNET) consumes one; the kernel also counts user namespaces created via unshare(CLONE_NEWUSER). Exhaustion yields EAGAIN.
• Bridge/ifindex limit – each network device (bridges, veth pairs, macvtap, etc.) receives an ifindex from a finite integer space. Stale bridges keep the ifindex occupied until fully removed.
• cgroup v2 limits – pids.max restricts total tasks (threads+processes) in the slice; memory.max caps RAM. Exceeding these causes fork() or malloc() failures, leading to silent container start‑up aborts.
• File descriptor limit – each netlink socket, bridge fd, and namespace bind mount consumes an FD. Low fs.file-max results in Too many open files errors in ip or docker calls.

Identifying Boundary Conditions

On the CI host we observed the following failure points:

1. Namespace exhaustion – after ~4800 namespaces, clab deploy returned Error: failed to create network namespace: Resource temporarily unavailable.
2. Ifindex depletion – after ~1.1M total interfaces (bridges + veth), ip link add returned RTNETLINK answers: Operation not permitted.
3. PID limit – a lab with 30 routers each launching 20 FRR threads hit pids.max=2048, causing clab to fail when launching the container’s init process (fork: Resource temporarily unavailable).
4. FD limit – fs.file-max=100000 was exceeded by the accumulation of netlink sockets from stale bridges, leading to docker: Error response from daemon: failed to create shim task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: fork/exec /usr/bin/docker-proxy: too many open files.

The laptop never reached these numbers because either the host was rebooted or the user manually ran a cleanup script after each test.

Mitigating Scaling Limitations

• Pre‑run cleanup – enforce removal of all Containerlab‑created bridges and namespaces before clab deploy.
• Limit topology size – split large labs into smaller sub‑labs that stay well under the limits.
• Increase host limits – where possible, raise /proc/sys/user/max_user_namespaces and fs.file-max via sysctl or the CI host’s provisioning.
• Adjust cgroup limits – request higher pids.max and memory.max for the CI runner’s slice, or run the job in an uncapped slice.
• Use ephemeral runners – configure the CI system to spin up fresh VMs or Docker‑in‑Docker containers per job, guaranteeing a clean slate.

Resolving the Issue

Updating Bridge Configuration

Add a cleanup step to the CI pipeline (example for GitLab CI):

# .gitlab-ci.yml
stages:
  - prepare
  - test

prepare:
  stage: prepare
  script:
    - ./cleanup-bridges.sh
    - ./cleanup-namespaces.sh
  when: always   # run even if previous stage failed

test:
  stage: test
  script:
    - clab deploy -t lab.clab.yml
    - ./run-tests.sh
    - clab destroy

The cleanup scripts are idempotent; ip link delete on a non‑existent bridge returns RTNETLINK answers: No such device.

Cleaning Leaked Namespaces

The cleanup-namespaces.sh script already removes all namespaces matching the Containerlab prefix. For extra safety, purge any namespace lacking a bind mount:

#!/usr/bin/env bash
# aggressive-namespace-cleanup.sh
set -euo pipefail

for ns in $(ip netns list); do
    if ! mountpoint -q /run/netns/$ns; then
        echo "Force‑removing orphaned namespace $ns"
        ip netns delete "$ns"
    fi
done

Adjusting Host Resource Limits

If the CI admin permits, raise limits via a privileged sysctl block in the runner’s config (example for a GitLab runner using Docker executor):

[[runners]]
  [...]
  [runners.docker]
    privileged = true
    volumes = ["/cache"]
    sysctls = [
      "user.max_user_namespaces=65536",
      "fs.file-max=200000",
    ]

Alternatively, adjust at runtime within the job (requires CAP_SYS_ADMIN):

sudo sysctl -w user.max_user_namespaces=65536
sudo sysctl -w fs.file-max=200000

Implementing Scalability Fixes

• Containerlab version – use clab >= 0.45.0 which adds the --cleanup flag to clab destroy to force removal of bridges and namespaces even if the container runtime fails.
• Labels for selective pruning – label bridges with containerlab=lab-name and prune only those:

for br in $(ip -br link show type bridge | awk '$2 ~ /containerlab=lab-name/ {print $1}'); do
    ip link set dev "$br" down
    ip link delete dev "$br" type bridge
done

By integrating pre‑run cleanup, monitoring key limits, and, where possible, raising those limits, the intermittent CI failures caused by stale bridges, leaked namespaces, and exhausted host resources can be eliminated.