From brittle netns shell to a clean lab harness

Introduction to Refactoring Ad‑Hoc Scripts

Ad‑hoc scripts for Linux network namespaces and veth pairs typically consist of a linear series of ip netns add, ip link add, and ip link set commands. They assume a clean host state, hard‑code interface names, and omit verification that objects exist after creation. When a step fails—e.g., the namespace already exists or a veth pair collides with an existing interface—the script either continues with stale state or aborts, leaving half‑created resources. The result is a brittle test harness that requires manual cleanup and makes CI runs non‑deterministic.

Importance of Idempotence in Test Harnesses

Idempotence guarantees that invoking the same operation multiple times yields the same end state. In a test harness this means:

Repeatability – a test can be run, fail, and be rerun without manual intervention.
Atomic recovery – a partial failure can be rolled back to a known clean state.
Predictable resource usage – no accumulation of orphaned namespaces or veth pairs across runs.
Simplified debugging – the harness state is always a function of the input topology description, not of historic script invocations.

Designing the Idempotent Test Harness

Determining Naming Rules for Consistency

Consistent naming eliminates collisions and makes introspection trivial. We adopt the following conventions:

Namespace name: test_ns_<role>_<idx> where <role> is a short identifier (e.g., client, server, router) and <idx> is a zero‑based integer unique within the role.
Veth pair name: veth_<nsA>_<nsB> where <nsA> and <nsB> are the full namespace names sorted lexicographically to guarantee the same name regardless of creation order.
Interface inside a namespace: inherits the veth side name (e.g., inside test_ns_client_0 the interface is veth_test_ns_client_0_test_ns_server_0).

These rules are deterministic, reversible, and easy to parse with simple shell utilities.

Implementing Topology Assertions for Validation

After creating the desired graph we run a set of assertions that confirm the operational state matches the intended topology:

Namespace existence – ip netns list contains every expected namespace.
Veth pair existence – ip link show lists each veth pair exactly once.
Endpoint placement – each veth side is enslaved to the correct namespace (ip -n <ns> link show <veth-side>).
Addressing (optional) – if IP addresses are assigned, verify with ip -n <ns> addr show <veth-side>.
Connectivity (optional) – a quick ping -c 1 -W 1 between peers to confirm L2/L3 viability.

Assertions are implemented as functions that return a non‑zero exit code on mismatch; the harness aborts and triggers cleanup.

Ensuring Deterministic Teardown for Reliability

Teardown must remove every resource the harness possibly created, even if the script exited mid‑flow. We achieve this by:

Tracking state – a temporary file ($STATE_DIR/created) records each namespace and veth pair as they are successfully created.
Atomic cleanup – on script exit (via trap) we read the state file in reverse order and delete veth pairs before namespaces (to avoid “device busy” errors).
Idempotent delete – each delete operation first checks existence (ip netns list | grep -q ^<ns>$) and only acts if present, making the cleanup safe to run multiple times.
Lockfile – a simple flock‑based lock (/var/run/test_harness.lock) prevents concurrent runs from interfering with each other’s state files.

Refactoring the Script

Converting Ad‑Hoc Script to Idempotent Operations

Replace linear ip netns add … blocks with functions that encapsulate the check‑then‑act pattern:

ns_ensure() {
    local ns=$1
    if ! ip netns list | grep -qx "$ns"; then
        ip netns add "$ns"
        echo "$ns" >>"$STATE_DIR/created"
    fi
}

Similarly for veth pairs:

veth_ensure() {
    local ns_a=$1 ns_b=$2
    local veth="veth_${ns_a}_${ns_b}"
    if ! ip link show "$veth" >/dev/null 2>&1; then
        ip link add "$veth" type veth peer name "veth_${ns_b}_${ns_a}"
        ip link set "$veth" netns "$ns_a"
        ip link set "veth_${ns_b}_${ns_a}" netns "$ns_b"
        echo "$veth" >>"$STATE_DIR/created"
    fi
}

Both functions are safe to call repeatedly; they only create when missing and record the resource for later teardown.

Integrating Topology Assertions and Naming Rules

A topology description can be kept as a simple associative array or a YAML/JSON file. For illustration we use a bash array:

declare -A TOPO=(
    ["client_0"]="server_0"
    ["server_0"]="client_0"
    ["router_0"]="client_1 server_1"
    ["client_1"]="router_0"
    ["server_1"]="router_0"
)

The harness iterates over the keys, builds namespace names via the naming rule, ensures namespaces, then ensures each veth pair only once (by checking that the pair has not already been recorded). After the creation loop we invoke assert_topology which walks the same data structure and validates the kernel state as described earlier.

Implementing Cleaner Recovery from Partial Failure

We wrap the main logic in a set -euo pipefail block and install a trap:

cleanup() {
    # Delete in reverse order to avoid dependencies
    if [[ -f "$STATE_DIR/created" ]]; then
        tac "$STATE_DIR/created" | while read -r line; do
            if [[ "$line" =~ ^veth_ ]]; then
                ip link delete "$line" 2>/dev/null || true
            else
                ip netns delete "$line" 2>/dev/null || true
            fi
        done
        rm -f "$STATE_DIR/created"
    fi
    rmdir "$STATE_DIR" 2>/dev/null || true
}
trap cleanup EXIT

If any command fails, the EXIT trap runs, reads the state file, and removes everything that was successfully created up to that point. Because the delete functions are idempotent, a second cleanup pass (e.g., manual rerun) is harmless.

Test Harness Implementation

Writing the Test Harness in a Scripting Language

We choose bash for its ubiquitous presence and direct access to ip netns commands. The harness consists of:

A configuration section (naming prefix, state directory, lockfile).
Helper functions (ns_ensure, veth_ensure, assert_topology, cleanup).
A main function that parses optional arguments (e.g., --topology-file) and drives the workflow.

The entire script is under 150 lines, making it easy to audit and embed in CI pipelines.

Example Code: Creating Namespaces and Veth Pairs

#!/usr/bin/env bash
set -euo pipefail

PREFIX="test_ns"
STATE_DIR=$(mktemp -d)
LOCKFILE="/var/run/test_harness.lock"

# ----- helpers ---------------------------------------------------------
ns_name() {
    local role=$1 idx=$2
    printf "%s_%s_%02d" "$PREFIX" "$role" "$idx"
}

veth_name() {
    local ns_a=$1 ns_b=$2
    # sort to guarantee same name irrespective of order
    if [[ "$ns_a" < "$ns_b" ]]; then
        printf "veth_%s_%s" "$ns_a" "$ns_b"
    else
        printf "veth_%s_%s" "$ns_b" "$ns_a"
    fi
}

ns_ensure() {
    local ns=$1
    if ! ip netns list | grep -qx "$ns"; then
        ip netns add "$ns"
        echo "$ns" >>"$STATE_DIR/created"
    fi
}

veth_ensure() {
    local ns_a=$1 ns_b=$2
    local veth=$(veth_name "$ns_a" "$ns_b")
    if ! ip link show "$veth" >/dev/null 2>&1; then
        ip link add "$veth" type veth peer name "veth_${ns_b}_${ns_a}"
        ip link set "$veth" netns "$ns_a"
        ip link set "veth_${ns_b}_${ns_a}" netns "$ns_b"
        echo "$veth" >>"$STATE_DIR/created"
    fi
}

# ----- topology assertion -----------------------------------------------
assert_topology() {
    local -n topo=$1   # nameref to associative array passed by caller
    for ns in "${!topo[@]}"; do
        # namespace existence
        if ! ip netns list | grep -qx "$ns"; then
            echo "ERROR: namespace $ns missing" >&2
            return 1
        fi
        # each expected neighbor must have a veth pair
        for neighbor in ${topo[$ns]}; do
            local veth=$(veth_name "$ns" "$neighbor")
            if ! ip link show "$veth" >/dev/null 2>&1; then
                echo "ERROR: veth $veth (between $ns and $neighbor) missing" >&2
                return 1
            fi
            # check that each side is in the correct netns
            local peer="veth_${neighbor}_${ns}"
            if ! ip -n "$ns" link show "$veth" >/dev/null 2>&1; then
                echo "ERROR: $veth not present in namespace $ns" >&2
                return 1
            fi
            if ! ip -n "$neighbor" link show "$peer" >/dev/null 2>&1; then
                echo "ERROR: $peer not present in namespace $neighbor" >&2
                return 1
            fi
        done
    done
    return 0
}

# ----- cleanup ---------------------------------------------------------
cleanup() {
    if [[ -f "$STATE_DIR/created" ]]; then
        tac "$STATE_DIR/created" | while read -r line; do
            if [[ "$line" =~ ^veth_ ]]; then
                ip link delete "$line" 2>/dev/null || true
            else
                ip netns delete "$line" 2>/dev/null || true
            fi
        done
        rm -f "$STATE_DIR/created"
    fi
    rmdir "$STATE_DIR" 2>/dev/null || true
}
trap cleanup EXIT

# ----- main -----------------------------------------------------------
main() {
    # acquire lock to prevent concurrent runs
    exec 200>"$LOCKFILE"
    flock -n 200 || { echo "Another instance is running; aborting." >&2; exit 1; }

    # Example static topology; could be loaded from a file
    declare -A TOPO=(
        ["client_0"]="server_0"
        ["server_0"]="client_0"
        ["router_0"]="client_1 server_1"
        ["client_1"]="router_0"
        ["server_1"]="router_0"
    )

    # Ensure namespaces
    for role in client server router; do
        for idx in 0 1; do   # adjust range as needed
            ns_ensure "$(ns_name "$role" "$idx")"
        done
    done

    # Ensure veth pairs (only once per edge)
    declared=()
    for ns in "${!TOPO[@]}"; do
        for neighbor in ${TOPO[$ns]}; do
            # normalize order to avoid duplicate work
            if [[ "$ns" < "$neighbor" ]]; then
                pair=("$ns" "$neighbor")
            else
                pair=("$neighbor" "$ns")
            fi
            # skip if already processed
            [[ " ${declared[*]} " =~ " ${pair[0]} ${pair[1]} " ]] && continue
            veth_ensure "${pair[0]}" "${pair[1]}"
            declared+=("${pair[0]} ${pair[1]}")
        done
    done

    # Validate
    assert_topology TOPO || {
        echo "Topology assertion failed" >&2
        exit 1
    }

    echo "Test harness ready. Namespaces and veth pairs created successfully."
    # Placeholder for user‑provided test logic
    # e.g., run a test suite here
}

main "$@"

Example Code: Asserting Topology and Cleaning Up

The assert_topology function shown above performs the validation. The cleanup trap guarantees that, on any exit path (including set -e triggered by a failed assertion), all tracked resources are removed. After a successful run the state file is empty; a subsequent run starts from a clean slate.

Troubleshooting Common Issues

Debugging Namespace and Veth Creation Failures

“File exists” – indicates a namespace or veth with the same name already present, possibly left over from a previous unclean run. Check with ip netns list and ip link show | grep veth_. Manual cleanup (ip netns del <ns>; ip link del <veth>) resolves it.
Permission denied – the script must be run with CAP_NET_ADMIN (typically as root or via sudo). Verify with id -nG and ensure the effective UID is 0.
No such device – occurs when trying to move a veth end into a namespace that hasn’t been created yet. Ensure ns_ensure is called before veth_ensure for both endpoints.

Handling Partial Failure and Recovery Scenarios

If the script dies midway through the veth creation loop, the state file contains only the resources successfully created. The EXIT trap runs cleanup, which reads the state file in reverse and deletes veth pairs before namespaces. Because each delete checks existence, a second pass is safe. To test this, you can insert return 1 or false at any point in the creation loop and observe that the trap removes everything created up to that point.

End of document.