LinkState
Go back

Reusable Harnesses Without Topology-Coupled Assertions

Introduction to Test Harness Design

Overview of Test Harness Requirements

A test harness provisions, configures, exercises, and validates network devices or software components in a repeatable manner. Core requirements include:

Importance of Portability in Test Harnesses

Portability eliminates the maintenance burden of maintaining multiple, near‑identical copies of a harness for each lab. When a harness assumes a fixed mapping such as “eth0 on router‑1 connects to eth1 on switch‑A”, any change in the underlying inventory forces a code change and risks introducing subtle bugs. Portable harnesses:

Design Principles for Portable Test Harnesses

Modularizing Setup and Verification Logic

Separate concerns into distinct layers:

  1. Orchestration layer – node lifecycle (power on/off, VM spawn, container start).
  2. Configuration layer – renders device‑specific configs from data models.
  3. Verification layer – protocol‑agnostic checkers (e.g., BGP state, ACL hit counts) that consume operational state via a unified API.
  4. Reporting layer – normalizes results into JUnit, TestResult, or custom JSON.

Each layer exposes a well‑defined interface (e.g., Python abstract base classes) so implementations can be swapped without affecting other layers.

Abstracting Interface Names and Node Order

Instead of referencing eth0 or ge-0/0/1 directly, the harness works with logical identifiers defined in an external topology file:

Node order is abstracted similarly: tests refer to nodes by role (e.g., leaf1, spine2) rather than by index in a list. The harness builds a runtime dictionary node_map[role] = node_object from the topology file.

Avoiding One-Off Assumptions and Hardcoding

PitfallPortable Alternative
Hardcoded IP subnets (e.g., 10.0.0.0/24)Subnet pools allocated dynamically from a CIDR block defined in the topology file.
Vendor‑specific CLI strings (e.g., configure terminal)Device driver abstractions that map generic commands (enter_config_mode) to vendor CLI/NETCONF/gNMI.
Fixed test sequence lengthData‑driven test steps where each step is a dictionary {action, args, expected} read from a test case file.
Assumed OS versionVersion‑agnostic feature detection (e.g., query show version and adapt parser).

By externalizing all variable data, the core harness logic remains unchanged across labs.

Architecture of a Portable Test Harness

High-Level Architecture Overview

+-------------------+      +-------------------+      +-------------------+
|   Orchestrator    |      |  Config Renderer  |      |   Verifier        |
| (node lifecycle) |<---->| (Jinja2/Templating)|<---->| (checkers)        |
+-------------------+      +-------------------+      +-------------------+
          ^                         ^                         ^
          |                         |                         |
          |        Topology & Test  |        Operational      |
          |        Data (YAML/JSON) |        State (gNMI/CLI) |
          +-------------------------+-------------------------+
                               +-------------------+
                               |   Reporter        |
                               | (JUnit, JSON, HTML)|
                               +-------------------+

Component-Based Design for Reusability

Each component is packaged as a Python module with a clear public API:

Dependencies are injected via constructor arguments or environment variables, enabling unit testing with mocks.

Data-Driven Approach for Configuration

All variable lives in data files:

The harness reads these files at startup, builds an internal representation, and proceeds through the test steps. Changing a lab only requires swapping the topology file; no code modification is needed.

Implementing Portable Test Harnesses

Using Parameterized Testing for Flexibility

Parameterized test frameworks (e.g., pytest.mark.parametrize, Robot Framework data‑driven tests) allow a single test function to iterate over a dataset. Example in Python with pytest:

import pytest

@pytest.mark.parametrize("role,expected_as", [
    ("leaf1", 65001),
    ("leaf2", 65002),
    ("leaf3", 65003),
])
def test_bgp_peer_established(orchestrator, verifier, role, expected_as):
    node = orchestrator.get_node(role)
    state = verifier.get_bgp_summary(node)
    assert state["local_as"] == expected_as
    assert state["peer_state"] == "Established"

The dataset (role,expected_as) can be generated dynamically from the topology file, eliminating hardcoded values.

Leveraging Templating for Interface and Node Configuration

Jinja2 (or similar) templates enable rendering of CLI or YANG‑based configs from abstract data. Example template for a loopback interface:

# templates/loopback.j2
interface {{ logical_iface.name }}
 description {{ logical_iface.description }}
 ip address {{ logical_iface.ip }}/{{ logical_iface.prefix }}

Rendering call:

from jinja2 import Environment, FileSystemLoader

env = Environment(loader=FileSystemLoader("templates"))
template = env.get_template("loopback.j2")
config = template.render(logical_iface=iface_data)

The iface_data dictionary contains only logical fields (name, description, ip, prefix). The mapping from logical name to physical interface (e.g., eth0, GigabitEthernet0/0/0) is performed by the orchestrator before sending the config to the device.

Example Code: Parameterizing Setup and Verification Logic

Below is a minimal, self‑contained example that demonstrates the full flow: reading topology, provisioning nodes (mocked), rendering configs, applying them, and verifying BGP state.

# portable_harness.py
import yaml
import jinja2
from abc import ABC, abstractmethod
from typing import Dict, Any

# ---------- Abstract Interfaces ----------
class NodeHandle(ABC):
    @abstractmethod
    def send_config(self, cfg: str) -> None: ...
    @abstractmethod
    def get_state(self, path: str) -> Any: ...

class Orchestrator(ABC):
    @abstractmethod
    def provision(self, topology: Dict) -> Dict[str, NodeHandle]: ...

class Verifier(ABC):
    @abstractmethod
    def check_bgp_summary(self, node: NodeHandle) -> Dict[str, Any]: ...

# ---------- Concrete Mock Implementations ----------
class MockNode(NodeHandle):
    def __init__(self, name: str):
        self.name = name
        self.configs = []
        self.state = {}

    def send_config(self, cfg: str) -> None:
        self.configs.append(cfg)
        # naive parsing for demo
        if "router bgp" in cfg:
            self.state["bgp_configured"] = True

    def get_state(self, path: str) -> Any:
        return self.state.get(path)

class MockOrchestrator(Orchestrator):
    def provision(self, topology: Dict) -> Dict[str, NodeHandle]:
        return {role: MockNode(name) for role, name in topology["nodes"].items()}

class MockVerifier(Verifier):
    def check_bgp_summary(self, node: NodeHandle) -> Dict[str, Any]:
        # In a real harness this would query gNMI/NETCONF
        return {
            "local_as": node.get_state("bgp_local_as") or 0,
            "peer_state": "Established" if node.get_state("bgp_configured") else "Idle",
        }

# ---------- Harness Core ----------
def load_topology(path: str) -> Dict:
    with open(path) as f:
        return yaml.safe_load(f)

def render_interface(node_role: str, iface_data: Dict) -> str:
    env = jinja2.Environment(loader=jinja2.FileSystemLoader("templates"))
    tmpl = env.get_template("interface.j2")
    return tmpl.render(node=node_role, iface=iface_data)

def apply_config(orch: Orchestrator, verifier: Verifier,
                 topology: Dict, node_role: str, iface_data: Dict):
    nodes = orch.provision(topology)
    node = nodes[node_role]
    cfg = render_interface(node_role, iface_data)
    node.send_config(cfg)
    # Simulate committing
    return node

def run_test(topology_path: str):
    topo = load_topology(topology_path)
    orch = MockOrchestrator()
    verifier = MockVerifier()

    for role, data in topo["interfaces"].items():
        node = apply_config(orch, verifier, topo, role, data)
        summary = verifier.check_bgp_summary(node)
        assert summary["peer_state"] == "Established", f"BGP not up on {role}"

if __name__ == "__main__":
    run_test("topology.yaml")

Explanation of portability aspects:

Troubleshooting Common Issues in Portable Test Harnesses

Debugging Interface and Node Configuration Issues

Resolving One-Off Assumption Violations

CLI Examples for Troubleshooting and Debugging

GoalCommandDescription
List all nodes and their resolved physical IDsharness-cli inventory list --lab lab-cShows mapping from logical role to VM/container name.
Verify template rendering matches device expectationsharness-cli template validate --template interface.j2 --schema interface_schema.jsonUses JSON‑Schema to ensure rendered output contains mandatory keys.
Run a single test step in isolationharness-cli test run --test-case bgp_peer.yaml --step 2 --dry-runExecutes only step 2, printing actions without touching devices.
Collect artifacts after a failureharness-cli artifacts collect --run-id 2024-09-24_15-30 --output ./artifactsPulls configs, logs, and telemetry from all nodes.

Scaling Portable Test Harnesses

Horizontal Scaling for Large-Scale Labs

Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email
Previous Post
Route policy intent versus forwarding outcome
Next Post
How much reordering becomes visible pain