Why Regression Baselines for a Network Copilot Must Be Versioned Alongside NOS Upgrades, Topology Changes, and Tool Adapter Releases

An engineering design review memo

Regression Baselines: Definition and Purpose

A regression baseline is an immutable snapshot of expected outputs from a deterministic set of network‑state queries (probes) that capture the known‑good behavior of a network under a specific configuration.

Typical probes include:

Baselines are stored as key‑value pairs or structured JSON/YAML files (one file per probe or a single aggregated manifest). When a change is introduced—NOS upgrade, topology re‑wire, or tool‑adapter update—the copilot re‑runs the same probes and compares the new outputs against the baseline. Deviations beyond a tolerated tolerance (e.g., timestamp fields, transient counters) trigger a regression alert.

Why we need baselines

1. Detect unintended side‑effects of planned changes before they affect production traffic.
2. Provide a trustworthy reference for the copilot’s natural‑language‑to‑tool translation layer, ensuring generated commands produce the expected observable state.
3. Enable automated gating in CI/CD pipelines: a change that fails the baseline comparison cannot be auto‑merged.

Why Versioning Baselines Is Essential

If baselines were kept as a single “latest” file, any upstream change (NOS, topology, tool) would silently overwrite the previous expectation, making it impossible to answer: “What was the expected behavior before change X?”

Versioning solves three concrete problems:

Thus, a regression baseline must be versioned in lock‑step with the three axes that can alter probe outputs: Network Operating System (NOS) release, network topology, and the set of tool adapters the copilot uses to execute probes.

Impact of NOS Upgrades

How NOS Versions Affect Probe Outputs

• Schema changes – New or deprecated fields in show command output (e.g., addition of lacp-rate in LACP status).
• Behavioral changes – Algorithms for ECMP load‑balancing, ARP timeout, or BGP route‑selection may shift, changing numeric values.
• Output formatting – Vendors sometimes alter column ordering, header text, or introduce JSON/YAML output modes.
• Feature toggles – New telemetry streams or counters appear only after a certain NOS version.

If a baseline was generated under NOS 1.0 and the network is upgraded to NOS 2.0 without updating the baseline, the copilot will flag every probe that touches a changed field as a regression, drowning real issues in noise.

Triple‑Tag Versioning Scheme

We encode the three axes in a single version string:

BASELINE_VERSION = <NOS_VERSION>.<TOPOLOGY_HASH>.<TOOL_ADAPTER_VERSION>

• NOS_VERSION – Semantic version string from the NOS release (e.g., 2.3.1).
• TOPOLOGY_HASH – SHA‑256 digest of a canonical topology representation (see Section 2).
• TOOL_ADAPTER_VERSION – Version of the adapter plugin that translates copilot intents to device‑specific CLI/API calls.

Each baseline lives in a Git‑tracked directory reflecting these tags:

baselines/
   nos-2.3.1/
      topo-<hash>/
         tool-adapter-3.1.0/
            baseline.json

When a NOS upgrade is planned, a baseline regeneration job runs against a lab clone of the network at the target NOS version, producing a new baseline under the new nos-<X.Y.Z> directory. The old baseline remains untouched, enabling rollback or comparative analysis.

Example: Upgrading from NOS 1.0 to NOS 2.0

Assume we have a baseline for NOS 1.0, topology hash a1b2c3…, tool adapter version 2.5.0.

Step 1 – Tag the NOS upgrade

git tag nos-v2.0.0
git push origin nos-v2.0.0

Step 2 – Provision a temporary lab (using your orchestrator, e.g., ArgoCD + Kubernetes)

kubectl apply -f lab-manifest-nos2.yaml   # deploys v2.0.0 images on virtual switches

Step 3 – Run the probe suite and capture output

#!/usr/bin/env bash
set -euo pipefail

NOS_VER="2.0.0"
TOPO_HASH=$(sha256sum topology.yaml | cut -d' ' -f1)
TOOL_VER="2.5.0"

OUTPUT_DIR="baselines/nos-${NOS_VER}/topo-${TOPO_HASH}/tool-adapter-${TOOL_VER}"
mkdir -p "${OUTPUT_DIR}"

for probe in $(cat probe-list.txt); do
    ./run-probe.sh "${probe}" > "${OUTPUT_DIR}/${probe}.json"
done

cat > "${OUTPUT_DIR}/manifest.json" <<EOF
{
  "nos_version": "${NOS_VER}",
  "topology_hash": "${TOPO_HASH}",
  "tool_adapter_version": "${TOOL_VER}",
  "generated_at": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
}
EOF

Step 4 – Commit and push

git add baselines/
git commit -m "Add baseline for NOS 2.0.0 (topo ${TOPO_HASH}, tool 2.5.0)"
git push origin main

Now any copilot query executed against a NOS 2.0.0 device is compared against the newly versioned baseline, while rollback to NOS 1.0 simply checks out the earlier baseline directory.

Effects of Topology Changes

How Topology Modifications Influence Probe Outputs

1. Device‑level changes – Adding/removing a leaf switch alters the set of interfaces visible in show interfaces status.
2. Path‑level changes – Modifying a link cost or adding a new LAG changes ECMP hashing, influencing show ip route (different next‑hop) and show bgp summary (different peer‑group advertisement counts).

If the baseline still reflects the old topology, the copilot interprets legitimate new paths as regressions. Conversely, an over‑generalized baseline (e.g., stripped of device‑specific identifiers) may miss real regressions caused by mis‑cabling.

Deriving a Topology Hash

We compute a SHA‑256 hash from a canonical, vendor‑neutral topology YAML that lists each node, its role, management IP, and interconnections. The hash is recomputed whenever the topology YAML changes.

Canonical topology YAML example

nodes:
  - name: leaf01
    role: leaf
    mgmt_ip: 10.0.0.1
    interfaces:
      - name: Ethernet1
        peer: spine01
        peer_if: Ethernet1
      - name: Ethernet2
        peer: spine02
        peer_if: Ethernet1
  - name: leaf02
    role: leaf
    mgmt_ip: 10.0.0.2
    interfaces:
      - name: Ethernet1
        peer: spine01
        peer_if: Ethernet2
      - name: Ethernet2
        peer: spine02
        peer_if: Ethernet2
  - name: spine01
    role: spine
    mgmt_ip: 10.0.0.10
    interfaces:
      - name: Ethernet1
        peer: leaf01
        peer_if: Ethernet1
      - name: Ethernet2
        peer: leaf02
        peer_if: Ethernet1
  - name: spine02
    role: spine
    mgmt_ip: 10.0.0.11
    interfaces:
      - name: Ethernet1
        peer: leaf01
        peer_if: Ethernet2
      - name: Ethernet2
        peer: leaf02
        peer_if: Ethernet2
links: []   # derived from interfaces; kept for readability

Hash computation:

TOPO_HASH=$(sha256sum topology.yaml | awk '{print $1}')

When the topology changes (e.g., adding leaf03), the YAML is edited, the hash changes, and a new baseline directory is created.

Automating Baseline Updates for Topology Changes

The following Python script is intended to be invoked from a CI pipeline whenever a topology PR is merged.

#!/usr/bin/env python3
import hashlib, json, os, subprocess, sys
from pathlib import Path

def file_sha256(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(8192), b""):
            h.update(chunk)
    return h.hexdigest()

def main():
    topo_file = Path(sys.argv[1])          # e.g., topology.yaml
    nos_ver   = os.getenv("NOS_VERSION")   # injected by CI
    tool_ver  = os.getenv("TOOL_ADAPTER_VERSION")

    if not topo_file.is_file():
        sys.exit(f"Topology file {topo_file} not found")

    topo_hash = file_sha256(topo_file)
    out_dir = Path(f"baselines/nos-{nos_ver}/topo-{topo_hash}/tool-adapter-{tool_ver}")
    out_dir.mkdir(parents=True, exist_ok=True)

    probe_list = Path("probe-list.txt").read_text().splitlines()

    for probe in probe_list:
        result = subprocess.run(
            ["./run-probe.sh", probe],
            capture_output=True,
            text=True,
            check=True,
        )
        (out_dir / f"{probe}.json").write_text(result.stdout)

    manifest = {
        "nos_version": nos_ver,
        "topology_hash": topo_hash,
        "tool_adapter_version": tool_ver,
        "generated_at": subprocess.check_output(
            ["date", "-u", +"%Y-%m-%dT%H:%M:%SZ"]
        ).decode().strip(),
    }
    (out_dir / "manifest.json").write_text(json.dumps(manifest, indent=2))

    print(f"Baseline written to {out_dir}")

if __name__ == "__main__":
    main()

CI usage (GitLab example)

baseline_topology:
  stage: test
  script:
    - export NOS_VERSION=$(cat NOS_VERSION)
    - export TOOL_ADAPTER_VERSION=$(cat TOOL_ADAPTER_VERSION)
    - python3 ./update_baseline_for_topology.py topology.yaml
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      changes:
        - topology.yaml

Influence of Tool Adapter Releases

How Tool Adapter Updates Affect Baselines

A tool adapter translates the copilot’s abstract intent (e.g., “get interface status for Ethernet1/1”) into device‑specific CLI, NETCONF/YANG, or REST calls. Adapter releases can affect baselines via:

• Command syntax changes – Adding support for a new CLI mode (e.g., show interfaces transceiver vs. legacy show interfaces).
• Response parsing differences – Switching from text‑parsing to structured JSON output changes the shape of captured data.
• Error handling – New retry logic or timeout values may cause transient differences (partial data vs. empty).
• Feature flags – Adapters may expose optional fields (like timestamp or sequence_number) that were previously omitted.

If the baseline was captured with adapter 2.4.0 and the copilot is upgraded to use adapter 3.1.0 without updating the baseline, the copilot will see a mismatch in every probe that now returns JSON instead of plain text, leading to false‑positive regressions.

Versioning Baselines with Tool Adapter Releases

We extend the version string to include the adapter version as the third component (see the triple‑tag scheme above). The adapter version is immutable once a baseline is generated; any adapter bump forces a new baseline generation.

At runtime, the copilot reads the manifest of the baseline it is about to compare against and aborts if the running adapter version does not match, preventing silent mismatches.

CLI Example: Updating Baselines with Tool Adapter Version 3.1

Assume we already have a baseline for NOS 2.0.0, topology hash d4e5f6…, and we are moving to tool adapter version 3.1.

#!/usr/bin/env bash
set -euo pipefail

NOS_VER="2.0.0"
TOPO_HASH="d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5"
TOOL_VER="3.1"

OUTPUT_DIR="baselines/nos-${NOS_VER}/topo-${TOPO_HASH}/tool-adapter-${TOOL_VER}"
mkdir -p "${OUTPUT_DIR}"

for probe in $(cat probe-list.txt); do
    ./run-probe.sh "${probe}" > "${OUTPUT_DIR}/${probe}.json"
done

cat > "${OUTPUT_DIR}/manifest.json" <<EOF
{
  "nos_version": "${NOS_VER}",
  "topology_hash": "${TOPO_HASH}",
  "tool_adapter_version": "${TOOL_VER}",
  "generated_at": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
}
EOF

git add baselines/
git commit -m "Add baseline for tool adapter 3.1 (NOS ${NOS_VER}, topo ${TOPO_HASH})"
git push origin main

Now the copilot, when running with adapter 3.1, will compare its outputs against this versioned baseline, ensuring that any observed differences stem from genuine network changes rather than adapter‑induced output variations.

Summary

Versioning regression baselines in tandem with NOS releases, topology changes, and tool‑adapter updates is essential for:

• Reliable rollback – restoring the exact pre‑change expectation.
• Auditability – providing a traceable record for compliance and reviews.
• Isolated experimentation – allowing parallel branches to validate against distinct baselines without interference.

By encoding the three axes into a deterministic directory structure and embedding the versions in a manifest, the network copilot gains a trustworthy, immutable reference that supports safe, automated change management.