CLI, graph diff, or LLM for namespace triage

Introduction

A hidden‑wiring namespace failure occurs when a Linux network namespace exists but its virtual interfaces, address assignments, or routing entries are missing or invisible to standard tooling. Symptoms include:

• ip netns list shows the namespace, yet ip netns exec <ns> ip link returns only the loopback device.
• Applications inside the namespace receive ENODEV or cannot communicate.
• Expected routes or NAT rules are absent, causing packet drops that are hard to trace because the namespace appears “empty” from the host view.

Such failures often stem from scripting errors, race conditions during interface moves, or cleanup scripts that delete a peer veth while leaving the namespace intact. Resolving them quickly reduces MTTR and prevents silent data‑plane black holes in containers, test environments, and service meshes.

Using iproute2 to Diagnose and Fix Namespace Failures

iproute2 Overview

The iproute2 suite (ip, ss, rtacct, …) talks directly to the kernel via netlink sockets, providing a low‑level, programmable interface suitable for scripting and automation.

Typical veth‑pair Workflow

# 1. Create namespaces
ip netns add ns-a
ip netns add ns-b

# 2. Create veth pair
ip link add veth-a type veth peer name veth-b

# 3. Move each end into its namespace
ip link set veth-a netns ns-a
ip link set veth-b netns ns-b

# 4. Bring interfaces up and assign addresses
ip netns exec ns-a ip link set dev veth-a up
ip netns exec ns-a ip addr add 10.0.0.1/24 dev veth-a
ip netns exec ns-b ip link set dev veth-b up
ip netns exec ns-b ip addr add 10.0.0.2/24 dev veth-b

# 5. Add default routes (if needed)
ip netns exec ns-a ip route add default via 10.0.0.2 dev veth-a
ip netns exec ns-b ip route add default via 10.0.0.1 dev veth-b

If any step fails silently (e.g., a typo in ip link set … netns …), the namespace may retain only lo, producing a hidden‑wiring state.

Topology Graph Diff for Failure Analysis

Concept

Model the network as a graph: nodes = namespaces, interfaces, bridges; edges = physical or virtual links (veth peers). By building a graph before and after a change and computing a structural diff, missing or extra edges/nodes become evident.

Building the Graph with Python/networkx

#!/usr/bin/env python3
import json, subprocess, networkx as nx

def ip_j(args):
    return json.loads(subprocess.check_output(["ip", "-j"] + args, text=True))

def build_graph():
    G = nx.Graph()
    # namespaces as nodes
    for ns in ip_j(["netns", "list"]):
        G.add_node(ns["name"], type="namespace")
    # interfaces
    for iface in ip_j(["link", "show"]):
        name = iface["ifname"]
        ns = iface.get("netns", {}).get("id") if "netns" in iface else None
        node_id = f"ns{ns}:{name}" if ns is not None else f"host:{name}"
        G.add_node(node_id, type="interface", netns=ns)
        # link to peer if present
        if "peer" in iface:
            peer = iface["peer"]
            peer_ns = iface.get("peer_netns", {}).get("id")
            peer_node = f"host:{peer}" if peer_ns is None else f"ns{peer_ns}:{peer}"
            G.add_edge(node_id, peer_node, type="veth")
    return G

if __name__ == "__main__":
    G_before = build_graph()
    nx.write_dot(G_before, "before.dot")
    # … perform changes …
    G_after = build_graph()
    nx.write_dot(G_after, "after.dot")

Detecting Missing Wiring

Convert DOT to simple edge lists and diff:

awk '/->/ {print $1, $3}' before.dot > before.edges
awk '/->/ {print $1, $3}' after.dot   > after.edges

echo "Missing edges:"
diff -u before.edges after.edges | grep '^-' | cut -c2-

echo "Added edges:"
diff -u before.edges after.edges | grep '^+' | cut -c2-

An absent edge such as host:veth-a -> ns1:veth-b signals that a veth peer was not moved into the expected namespace—the classic hidden‑wiring symptom.

AI‑Assisted Workflow for Rapid Resolution

Overview

A large language model (LLM) interprets observed symptoms (e.g., empty ip link inside a namespace) and suggests corrective iproute2 commands. The model does not execute commands; it acts as an advisory tool that narrows the operator’s search space.

Workflow Steps

1. Anomaly detection – flag namespaces with fewer than two interfaces (loopback + expected veth).

2. Evidence collection – gather:

   • ip netns exec <ns> ip link show
   • Current topology graph (JSON) for the suspect namespace
   • Recent orchestrator/log entries (e.g., API calls that created a veth pair)

3. Prompt the LLM (temperature = 0 for determinism) with a structured request, e.g.:

   You are a network engineer. The namespace "ns-a" shows only the loopback interface.
   Current topology (JSON):
   {
     "nodes": [...],
     "edges": [...]
   }
   Recent orchestrator action: added veth pair veth-x/veth-y and attempted to move veth-y into ns-a.
   List the exact iproute2 commands needed to repair the namespace, assuming the peer veth-x remains in the host namespace.

4. Receive suggestion – e.g.:

   ip link set veth-y netns ns-a
   ip netns exec ns-a ip link set dev veth-y up
   ip netns exec ns-a ip addr add 10.0.0.2/24 dev veth-y
   ip netns exec ns-a ip route add default via 10.0.0.1 dev veth-y

5. Validate – run a dry‑run (ip -netns ns-a link show) or apply in a sandbox before committing to production.
   Important: Review the LLM output; it may hallucinate names or addresses. The operator retains final authority.

Troubleshooting with iproute2 and Graph Diff

Common Symptoms and Fixes

Systematic iproute2 Checks

NS=problematic
echo "=== Interfaces in $NS ==="
ip netns exec $NS ip link show
echo "=== Addresses ==="
ip netns exec $NS ip -br addr show
echo "=== Routes ==="
ip netns exec $NS ip route show

Analyzing Graph Diff for the Suspect Namespace

grep "ns-$NS" after.edges | while read a b; do
    if ! grep -q "$a $b" before.edges; then
        echo "NEW edge: $a -- $b"
    fi
done

Missing edges pinpoint the exact veth that failed to migrate.

Code and CLI Examples

iproute2 Command Snippets

# Create and list namespaces
ip netns add demo
ip netns list
ip netns exec demo ip link show
ip netns delete demo

# Configure a veth pair
ip link add veth0 type veth peer name veth1
ip link set veth1 netns demo
ip netns exec demo ip addr add 192.168.100.2/24 dev veth1
ip netns exec demo ip link set veth1 up
ip addr add 192.168.100.1/24 dev veth0
ip link set veth0 up
ip netns exec demo ping -c 3 192.168.100.1

Graph‑Diff Helper Script (build_topology.py)

#!/usr/bin/env python3
import json, subprocess, networkx as nx, sys

def ip_j(args):
    return json.loads(subprocess.check_output(["ip", "-j"] + args, text=True))

def main():
    G = nx.Graph()
    for ns in ip_j(["netns", "list"]):
        G.add_node(ns["name"], kind="ns")
    for link in ip_j(["link", "show"]):
        name = link["ifname"]
        ns = link.get("netns", {}).get("id")
        node = f"host:{name}" if ns is None else f"ns{ns}:{name}"
        G.add_node(node, kind="iface", netns=ns)
        if "peer" in link:
            peer = link["peer"]
            peer_ns = link.get("peer_netns", {}).get("id")
            peer_node = f"host:{peer}" if peer_ns is None else f"ns{peer_ns}:{peer}"
            G.add_edge(node, peer_node, kind="veth")
    nx.write_dot(G, sys.argv[1])

if __name__ == "__main__":
    main()

Usage

python3 build_topology.py before.dot   # before changes
# … apply changes …
python3 build_topology.py after.dot    # after changes

Edge‑list diff

awk '/->/ {print $1, $3}' before.dot > before.edges
awk '/->/ {print $1, $3}' after.dot   > after.edges

echo "Missing edges:"
diff -u before.edges after.edges | grep '^-' | cut -c2-

echo "Added edges:"
diff -u before.edges after.edges | grep '^+' | cut -c2-

Scaling Limitations and Considerations

iproute2

• Each command triggers a netlink round‑trip; configuring thousands of interfaces yields O(N) syscall overhead.
• Batching via ip -batch <file> reduces per‑command latency but does not eliminate the linear syscall count.

Graph Diff

• Building the graph scans all links and namespaces: O(L + N).
• Diffing is a set‑based comparison, also linear.
• Memory usage scales with nodes/edges; for ~10⁵ interfaces a few hundred MB of RAM is typical and manageable on a control node.

AI‑Assisted Workflow Limits

• Context window – LLMs accept only 8k–32k tokens; a full data‑center topology may exceed this, requiring summarization or scoping to the suspect subsystem.
• Token cost – Large prompts increase expense; frequent AI queries in large environments can become economically non‑trivial.
• Optimization – Cache successful command patterns for recurring symptoms and reuse them to avoid repeated LLM calls.

Final note: Combining deterministic iproute2 checks, topology‑graph diffs, and an AI‑assisted suggestion loop yields fast, reproducible diagnosis while preserving operator authority over the final remediation.