Intent pipelines need graph checks before ip link

Introduction to Topology-as-Code Pipeline

Topology‑as‑Code (TaC) treats the desired network fabric—namespaces, virtual Ethernet (veth) pairs, and TUN devices—as declarative artifacts stored in version‑controlled files. A pipeline reads these artifacts, renders an in‑memory graph that represents the intended topology, validates that the graph matches the current kernel state, and only then applies changes. By separating render (what we intend) from apply (what we effect) and inserting a verification gate, the pipeline guarantees that any subsequent packet‑level test operates on a known, reproducible topology.

Benefits

• Determinism – Identical configuration always yields the same graph, eliminating drift from manual ip commands.
• Safety – The verification step blocks application when the rendered graph diverges from the applied state, preventing accidental overwrites or dangling interfaces.
• Auditability – Every change is a diff in a Git repository; the pipeline logs pre‑ and post‑state graphs for compliance.
• Reusability – Graph rendering libraries (e.g., NetworkX, graphviz) can produce visual diagrams for documentation or review.
• Scalability – The pipeline can be parallelized across workers handling disjoint sub‑graphs, provided the verification gate remains atomic per worker.

Designing the Pipeline Architecture

Namespace Management

Creating and Managing Namespaces
Namespaces are created with ip netns add <name> and removed with ip netns del <name>. Desired namespaces are stored in YAML:

namespaces:
  - name: ns-left
    sysctls:
      net.ipv4.ip_forward: 1
  - name: ns-right
    sysctls: {}

Configuring Namespace Settings
After creation, sysctl values are set inside the namespace via ip netns exec <ns> sysctl -w <key>=<value>. The pipeline applies each entry under sysctls sequentially, recording success/failure for rollback.

Virtual Ethernet (veth) Pair Management

Creating and Managing veth Pairs
A veth pair is created with ip link add <veth-a> type veth peer name <veth-b>. Endpoints are moved into namespaces with ip link set <veth-a> netns <ns-a> and similarly for <veth-b>. YAML representation:

veth_pairs:
  - left: ns-left
    right: ns-right
    left_iface: veth-l
    right_iface: veth-r
    mtu: 1500

Configuring veth Pair Settings
Post‑creation, the pipeline sets MTU, link state (up/down), and optional ethtool features (e.g., ethtool -K <iface> tso off). All parameters are declared per‑pair in the YAML.

TUN Device Management

Creating and Managing TUN Devices
TUN devices are created via ip tuntap (requires CAP_NET_ADMIN):

ip tuntap add dev tun0 mode tun user $(whoami)

The device is then placed into a namespace and assigned an IP address. YAML snippet:

tun_devices:
  - name: tun0
    namespace: ns-left
    address: 10.0.0.1/24
    mtu: 1500

Configuring TUN Device Settings
After creation, the pipeline sets the interface MTU, brings it up, and optionally configures queue length (txqueuelen) or BPF filters via tc.

Implementing the Pipeline

Rendering Graph Structure

The pipeline loads the YAML, builds a NetworkX MultiGraph where nodes are namespaces and edges represent veth pairs; TUN devices are modeled as pendant edges attached to a namespace node with a special type=tun attribute.

import yaml, networkx as nx

def load_topology(path):
    with open(path) as f:
        data = yaml.safe_load(f)
    G = nx.MultiGraph()
    for ns in data.get('namespaces', []):
        G.add_node(ns['name'], type='namespace')
    for pair in data.get('veth_pairs', []):
        G.add_edge(pair['left'], pair['right'],
                   left_iface=pair['left_iface'],
                   right_iface=pair['right_iface'],
                   mtu=pair.get('mtu', 1500),
                   type='veth')
    for tun in data.get('tun_devices', []):
        G.add_edge(tun['namespace'], f"tun:{tun['name']}",
                   iface=tun['name'],
                   address=tun.get('address'),
                   mtu=tun.get('mtu', 1500),
                   type='tun')
    return G

The rendered graph is serialized to DOT for visual review (nx.drawing.nx_agraph.write_dot(G, 'desired.dot')) and hashed (SHA‑256) to serve as the desired state fingerprint.

Verifying Rendered Graph Structure

The pipeline queries the kernel for the actual state:

• Namespaces – ip netns list → set of strings.
• Veth pairs – ip -d link show type veth → parse each line for ifname, peer name, and netns.
• TUN devices – ip link show type tun → parse ifname, netns, MTU.

From this data a second NetworkX graph (actual) is constructed using the same node/edge semantics. The pipeline then computes differences:

def graph_diff(desired, actual):
    node_diff = set(desired.nodes) ^ set(actual.nodes)
    edge_diff = []
    for u, v, k, data in desired.edges(keys=True, data=True):
        if not actual.has_edge(u, v, k):
            edge_diff.append(('missing', (u, v, k, data)))
        else:
            actual_data = actual.get_edge_data(u, v, k)
            if data != actual_data:
                edge_diff.append(('mismatch', (u, v, k, data, actual_data)))
    for u, v, k, data in actual.edges(keys=True, data=True):
        if not desired.has_edge(u, v, k):
            edge_diff.append(('extra', (u, v, k, data)))
    return node_diff, edge_diff

If either node_diff or edge_diff is non‑empty, verification fails.

Handling Mismatches and Errors
On failure the pipeline:

1. Logs the diff in JSON to verification-failure.json.
2. Emits a Prometheus metric tac_verification_failed_total{reason="node_or_edge_mismatch"}.
3. Aborts the apply stage and returns a non‑zero exit code, prompting the CI system to halt the rollout.
4. Optionally triggers an automated rollback to the last known good state (stored as a Git tag).

Applying Configuration

When verification passes, the pipeline executes the apply phase in a deterministic order:

1. Namespaces – create missing namespaces, delete extra ones (if prune:true), apply sysctls.
2. Veth pairs – create missing pairs, move endpoints, set MTU, bring up.
3. TUN devices – create missing TUN, assign IP, set MTU, bring up.

Each step is wrapped in a try/except block; on error the pipeline records the failing resource, attempts a compensating action (e.g., delete a half‑created veth pair), and then aborts.

Handling Configuration Errors and Conflicts
Conflicts arise when two definitions attempt to create the same interface name in the same namespace. The pipeline detects this during the render phase by checking for duplicate (namespace, ifname) tuples and fails fast with a clear error message:
Conflict: interface veth0 already defined in ns-left.

Troubleshooting the Pipeline

Common Issues and Errors

• Namespace already exists with different sysctl – the pipeline will not mutate sysctl of an existing namespace unless force:true is set; otherwise it reports a conflict.
• veth peer already moved – attempting ip link set <iface> netns <ns> when the peer is already in another namespace yields RTNETLINK answers: File exists. The pipeline catches this and suggests using ip link delete <iface> first.
• Missing CAP_NET_ADMIN – ip tuntap add fails with Operation not permitted. The pipeline checks the effective UID/capabilities before invoking the command.
• Address overlap – if two TUN devices receive overlapping subnets, the pipeline flags it during verification by comparing CIDR blocks.

Debugging Techniques

• Structured logging – each pipeline run emits a JSON log line with fields: run_id, timestamp, stage, status, resource_type, resource_name, message.
• Prometheus metrics – counters for tac_namespaces_created_total, tac_veth_created_total, tac_tun_created_total, and histograms for stage latency.
• Grafana dashboard – visualizes the desired vs. actual graph diff over time.
• tac-validate subcommand – loads YAML, runs the render step, and prints the DOT representation; useful for catching syntax errors before execution.
• tac-diff subcommand – compares the current state (gathered via ip commands) against the last successful apply, outputting a unified diff of interface additions/deletions.

Code Examples and CLI Commands

Creating and Managing Namespaces

# Create namespace
ip netns add ns-left

# List namespaces
ip netns list

# Set sysctl inside namespace
ip netns exec ns-left sysctl -w net.ipv4.ip_forward=1

# Remove namespace (only if no interfaces remain)
ip netns del ns-left

Creating and Managing veth Pairs

# Create veth pair
ip link add veth-l type veth peer name veth-r

# Move ends into namespaces
ip link set veth-l netns ns-left
ip link set veth-r netns ns-right

# Configure MTU and bring up
ip netns exec ns-left ip link set dev veth-l mtu 9000 up
ip netns exec ns-right ip link set dev veth-r mtu 9000 up

# Verify peer relationship
ip -d link show veth-l netns ns-left
ip -d link show veth-r netns ns-right

# Delete pair
ip link delete veth-l type veth

Creating and Managing TUN Devices

# Create TUN device (requires CAP_NET_ADMIN)
ip tuntap add dev tun0 mode tun user $(whoami)

# Move into namespace
ip link set tun0 netns ns-left

# Assign address and bring up
ip netns exec ns-left ip addr add 10.0.0.1/24 dev tun0
ip netns exec ns-left ip link set dev tun0 up

# Set MTU (optional)
ip netns exec ns-left ip link set dev tun0 mtu 1400

# Remove TUN device
ip link delete tun0 type tun

Example Code for Rendering and Verifying Graph Structure

#!/usr/bin/env python3
import subprocess, yaml, networkx as nx, hashlib, json, sys

def run_ip_netns():
    out = subprocess.check_output(['ip', 'netns', 'list'], text=True)
    return {line.split()[0] for line in out.strip().splitlines() if line}

def run_ip_link(kind):
    return subprocess.check_output(['ip', '-d', 'link', 'show', 'type', kind], text=True)

def parse_veth(link_out):
    edges = []
    for line in link_out.splitlines():
        if not line.strip():
            continue
        if '@' not in line:
            continue
        iface = line.split(':')[1].strip().split('@')[0]
        peer = line.split('@')[1].split(':')[0].strip()
        netns = None
        if 'netns' in line:
            netns = line.split('netns')[1].strip().split()[0]
        edges.append((iface, peer, netns))
    return edges

def parse_tun(link_out):
    edges = []
    for line in link_out.splitlines():
        if not line.strip():
            continue
        iface = line.split(':')[1].strip()
        netns = None
        if 'netns' in line:
            netns = line.split('netns')[1].strip().split()[0]
        mtu = None
        for part in line.split():
            if part.startswith('mtu'):
                mtu = int(part.split('=')[1]) if '=' in part else int(part.split()[1])
        edges.append((iface, netns, mtu))
    return edges

def build_actual_graph():
    G = nx.MultiGraph()
    # namespaces
    for ns in run_ip_netns():
        G.add_node(ns, type='namespace')
    # veth pairs
    veth_out = run_ip_link('veth')
    for iface, peer, netns in parse_veth(veth_out):
        ns_a = netns if netns else 'default'
        ns_b = netns if netns else 'default'  # both ends in same ns if netns present
        # Determine peer namespace: if the peer appears in another line with its own netns,
        # we would need a second pass; for simplicity we assume symmetric placement.
        G.add_edge(ns_a, ns_b,
                   left_iface=iface,
                   right_iface=peer,
                   mtu=1500,
                   type='veth')
    # TUN devices
    tun_out = run_ip_link('tun')
    for iface, netns, mtu in parse_tun(tun_out):
        ns = netns if netns else 'default'
        G.add_edge(ns, f"tun:{iface}",
                   iface=iface,
                   address=None,
                   mtu=mtu if mtu else 1500,
                   type='tun')
    return G

def main():
    if len(sys.argv) < 2:
        print("Usage: tac-verify <topology.yaml>")
        sys.exit(1)
    desired = load_topology(sys.argv[1])
    actual = build_actual_graph()
    node_diff, edge_diff = graph_diff(desired, actual)
    if not node_diff and not edge_diff:
        print("Verification passed.")
        sys.exit(0)
    else:
        print("Verification failed.")
        print("Node diff:", node_diff)
        print("Edge diff:", edge_diff)
        with open('verification-failure.json', 'w') as f:
            json.dump({
                'node_diff': list(node_diff),
                'edge_diff': edge_diff
            }, f, indent=2)
        sys.exit(1)

if __name__ == '__main__':
    main()

This script demonstrates the full render‑verify cycle: load the desired topology from YAML, construct the actual graph from the kernel, compare the two, and report any mismatches. Integrating it into a CI/CD pipeline ensures that only verified, packet‑test‑ready topologies are promoted.