Investigating CI/CD Pipeline Inconsistencies for Containerlab Labs

Introduction

On a developer workstation you typically run Containerlab with the default Docker daemon, unrestricted cgroups, and a host network that allows bridge, macvlan, or ipvlan interfaces without extra privileges. In CI (GitLab, GitHub Actions, Azure Pipelines, etc.) the job runs inside a container managed by a runner executor (docker, kubernetes, shell, or virtual machine). The runner imposes its own security profile, resource quotas, and network isolation, which often diverge from the local developer’s setup.

The most frequent failure modes when a lab that passes locally is re‑run in CI are:

1. Missing Linux capabilities (e.g., NET_ADMIN, SYS_PTRACE) required by FRR or Containerlab to create namespaces, bind to privileged ports, or load kernel modules.
2. Nested‑network assumptions – the lab expects to create a Docker bridge network inside the job container, but the runner’s Docker daemon may be disabled, run in --userns-remap mode, or be blocked by AppArmor/seccomp profiles.
3. cgroup limits – the runner may constrain memory, CPU, or pids via its own cgroup, causing OOM kills or fork failures when Containerlab spawns many veth pairs.
4. Host defaults – sysctl values such as net.ipv4.ip_forward, net.ipv6.conf.all.disable_ipv6, or fs.inotify.max_user_instances differ between the laptop and the CI image, breaking routing or FRR startup.

The following sections walk through a disciplined investigation that starts from an observed failure, isolates the root cause, and ends with a repeatable preflight contract that can be run in any CI environment.

Investigating Runner Capabilities

Understanding Runner Configurations

Most CI runners launch a privileged container for the job, but “privileged” does not guarantee all capabilities are retained. For example, the GitLab Docker executor drops CAP_SYS_MODULE and CAP_SYS_RAWIO by default, while the docker executor in GitHub Actions runs with --cap-add=ALL only when you explicitly set privileged: true.

To see what the runner actually provides, run inside the job container:

# List effective capabilities
capsh --print | grep -E "CapEff|CapPerm"

# Or, if capsh is not installed:
grep CapEff /proc/$$/status

Typical output on a restricted GitLab runner:

CapEff: 0000000000000000
CapPerm: 0000000000000000

indicating that no capabilities are retained – the container is running with a full capability drop.

Identifying Key Differences

On a laptop you can verify the daemon’s default capabilities:

docker run --rm --cap-add=ALL alpine capsh --print | grep CapEff

which usually shows a non‑zero mask (e.g., 0000000fffffffff). The delta between the local mask and the CI mask tells you which capabilities you must explicitly add.

Example: Comparing Docker Runner Configurations

Add this snippet to a GitLab CI job to dump the runner’s Docker daemon info and compare it to a local baseline:

# .gitlab-ci.yml
debug_runner:
  image: docker:latest
  services:
    - docker:dind
  script:
    - docker info | grep -E "Capabilities|Cgroup Driver|Kernel Version"
    - docker run --rm --privileged alpine capsh --print | grep CapEff

Run the same commands on your laptop (outside any CI container) and note the differences. If the CI output shows Capabilities: [] or a missing NET_ADMIN, you know you must add it via the runner configuration:

GitLab

debug_runner:
  tags: [docker]
  variables:
    DOCKER_DRIVER: overlay2
  before_script:
    - docker run --rm --cap-add=NET_ADMIN --cap-add=SYS_PTRACE alpine true

GitHub Actions

jobs:
  test:
    runs-on: ubuntu-latest
    container:
      options: --cap-add=NET_ADMIN --cap-add=SYS_PTRACE

Analyzing Nested‑Network Assumptions

Network Topology and Isolation in CI Environments

Containerlab creates a management network (usually a Docker bridge) to interconnect the lab nodes and the host. In CI, the job container often shares the runner’s Docker daemon via a socket (/var/run/docker.sock). However, many runners enable user namespace remapping (--userns-remap=default) or run the daemon inside a restricted container (e.g., Docker-in-Docker with --privileged but without allowing docker network create).

When the lab tries to create a bridge network, the daemon may return:

Error response from daemon: user specified IP address is supported only when connecting to networks with user configured subnets

or simply:

Error: failed to create endpoint ...: network not found

Troubleshooting Network‑Related Issues

Start by checking whether the Docker daemon inside the job can list networks:

docker network ls

If the list is empty or only shows the default bridge, host, and none, the daemon is likely unable to create new networks. Verify the daemon’s configuration:

docker info | grep -E "Network Drivers|Default Address Pools|Userland Proxy"

Look for userns-remap in the output; if present, the daemon maps container UIDs to a subordinate range, which prevents the lab’s containers from accessing host‑mounted sockets or creating VLANs that rely on the host’s uid/gid mapping.

Code Example: Validating Network Creation

#!/usr/bin/env bash
set -euo pipefail

NETLAB="ci-mgmt"
if ! docker network inspect "$NETLAB" >/dev/null 2>&1; then
  echo "Creating management network $NETLAB"
  docker network create \
    --driver bridge \
    --subnet 172.20.0.0/16 \
    --gateway 172.20.0.1 \
    "$NETLAB"
else
  echo "Network $NETLAB already exists"
fi

# Verify connectivity
docker run --rm --net "$NETLAB" alpine ping -c 1 172.20.0.1

If the docker network create call fails with permission denied or operation not supported, you have identified a runner‑level restriction. The fix is either to disable user namespace remapping in the runner config or to run the job with --privileged and --security-opt apparmor=unconfined (only if your security policy allows it).

Examining cgroup Limits and Host Defaults

Introduction to cgroup Limits

CI runners frequently impose cgroup v1 limits on memory (memory.limit_in_bytes), CPU (cpu.cfs_quota_us), and maximum number of processes (pids.max). Containerlab spawns a process per node (FRR, nginx, etc.) plus a veth pair for each link; a lab with 20 nodes and 30 links can easily exceed a low pids.max of 512, resulting in fork: Resource temporarily unavailable.

Memory limits cause OOM kills when FRR allocates large routing tables or when the lab uses large PCAP captures.

Identifying and Adjusting Host Defaults

First, inspect the cgroup hierarchy of the job’s PID 1:

cat /proc/$$/cgroup

Typical output on a GitLab runner using the docker executor:

0::/builds/group/project/_builds/0/script

Then check the actual limits:

# Memory limit (in bytes)
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
# CPU quota (microseconds per period)
cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
# PID limit
cat /sys/fs/cgroup/pids/pids.max

Compare these to the values on your laptop (usually max or -1 for unlimited).

If you find a restrictive pids.max, you can raise it only if the runner permits writing to the cgroup filesystem (many do not). In that case you must request a larger limit from the platform admin or split the lab into smaller sub‑labs.

Host sysctl defaults also matter. Commonly needed values for Containerlab/FRR:

# Enable IPv4 forwarding
sysctl net.ipv4.ip_forward=1
# Enable IPv6 forwarding
sysctl net.ipv6.conf.all.forwarding=1
# Disable IPv6 autoconf on interfaces (prevents duplicate addresses)
sysctl net.ipv6.conf.all.accept_ra=0
# Increase inotify watches (used by FRR’s watchdog)
sysctl fs.inotify.max_user_instances=512
sysctl fs.inotify.max_user_watches=1048576

On a fresh CI image these may be set to conservative values (e.g., net.ipv4.ip_forward=0).

CLI Example: Managing cgroups (if allowed)

If the runner allows you to create a child cgroup, you can apply custom limits before launching Containerlab:

# Create a cgroup under the existing memory controller
sudo cgcreate -g memory:/ci-lab
# Set a 2 GiB memory ceiling
sudo cgset -r memory.limit_in_bytes=$((2*1024*1024*1024)) /ci-lab
# Move the current shell into the cgroup
sudo cgclassify -g memory:/ci-lab $$
# Verify
cat /sys/fs/cgroup/memory/ci-lab/memory.limit_in_bytes

Caveat: Many CI environments mount the cgroup filesystem as read‑only for jobs, rendering cgcreate/cgset ineffective. In such cases you must rely on the runner’s built‑in resource settings (e.g., GitLab’s memory_limit variable or GitHub Actions’ resources: block).

Establishing a Preflight Contract

Defining a Repeatable Preflight Checklist

A preflight contract is a set of idempotent checks that must return success before the lab is instantiated. The checklist should cover:

1. Capabilities – required NET_ADMIN, SYS_PTRACE, optionally SYS_MODULE.
2. Docker daemon accessibility – ability to docker info and create a bridge network.
3. cgroup limits – memory ≥ 1 GiB, pids.max ≥ 1024 (or as dictated by lab size).
4. Sysctl defaults – net.ipv4.ip_forward=1, net.ipv6.conf.all.forwarding=1, fs.inotify.max_user_instances≥256.
5. Available storage – at least 2 GiB free in /var/lib/docker or the overlay workspace.

Each check returns a non‑zero exit code on failure, causing the CI job to abort early.

Implementing Automated Preflight Checks

Below is a portable Bash script (ci-preflight.sh) that can be sourced at the start of any job:

#!/usr/bin/env bash
set -euo pipefail

# ---------- 1. Capabilities ----------
required_caps=("NET_ADMIN" "SYS_PTRACE")
for cap in "${required_caps[@]}"; do
  if ! capsh --print | grep -qE "CapEff.*${cap}"; then
    echo "ERROR: Missing required capability: $cap"
    exit 1
  fi
done

# ---------- 2. Docker daemon & network ----------
if ! docker info >/dev/null 2>&1; then
  echo "ERROR: Cannot contact Docker daemon"
  exit 1
fi

NETLAB="ci-mgmt"
if ! docker network inspect "$NETLAB" >/dev/null 2>&1; then
  echo "INFO: Creating management network $NETLAB"
  if ! docker network create --driver bridge --subnet 172.20.0.0/16 --gateway 172.20.0.1 "$NETLAB"; then
    echo "ERROR: Failed to create Docker network"
    exit 1
  fi
fi

# ---------- 3. cgroup limits ----------
mem_limit=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes)
if [[ "$mem_limit" -ne -1 && "$mem_limit" -lt $((1024*1024*1024)) ]]; then
  echo "ERROR: Memory limit too low: $mem_limit bytes"
  exit 1
fi

pids_max=$(cat /sys/fs/cgroup/pids/pids.max)
if [[ "$pids_max" -ne -1 && "$pids_max" -lt 1024 ]]; then
  echo "ERROR: pids.max too low: $pids_max"
  exit 1
fi

# ---------- 4. Sysctl defaults ----------
declare -A sysctl_req=(
  ["net.ipv4.ip_forward"]=1
  ["net.ipv6.conf.all.forwarding"]=1
  ["fs.inotify.max_user_instances"]=256
)
for key in "${!sysctl_req[@]}"; do
  val=$(sysctl -n "$key")
  if [[ "$val" -lt "${sysctl_req[$key]}" ]]; then
    echo "ERROR: Sysctl $key=$val, expected >= ${sysctl_req[$key]}"
    exit 1
  fi
done

# ---------- 5. Disk space ----------
docker_root=$(docker info --format '{{.DockerRootDir}}')
free=$(df --output=avail "$docker_root" | tail -1)
if [[ "$free" -lt $((2*1024*1024*1024)) ]]; then
  echo "ERROR: Insufficient disk space in $docker_root: $free KB"
  exit 1
fi

echo "PREFLIGHT PASSED"
exit 0

Add it to your CI pipeline:

# .gitlab-ci.yml
lab_test:
  image: containerlab/srl:latest
  script:
    - chmod +x ci-preflight.sh
    - ./ci-preflight.sh
    - containerlab deploy -t lab.clab.yml
    - containerlab destroy -t lab.clab.yml

If any check fails, the job stops before containerlab deploy consumes time or resources.

The script is deliberately defensive: it uses set -euo pipefail, checks each condition independently, and prints a clear error message. You can extend it with version checks (e.g., verify FRR version) or additional lab‑specific validations as needed.