Configuration Generation Failures: Diagnosis Guide

When a generated configuration is incorrect, the failure typically stems from one of four orthogonal sources:

1. Underspecified operator intent – missing or ambiguous constraints in the intent model.
2. Stale topology facts – outdated device, interface, or neighbor information.
3. Missing policy context – required QoS, security, or routing policies not supplied to the renderer.
4. Bad renderer – defects in the template or code that maps intent + facts + policy to device‑specific configuration.

Each hypothesis requires a distinct diagnostic test. The sections below provide concrete CLI commands, code snippets, and validation strategies to isolate the root cause.

Identifying Potential Failure Causes

Start from the observable symptom and follow this decision tree:

Troubleshooting Underspecified Operator Intent

Analyzing Intent Specifications

Intent is usually stored as a structured document (YANG instance). Validation involves:

1. Schema compliance – does the instance conform to the YANG module?
2. Semantic completeness – are all mandatory leafs present and within allowed ranges?
3. Ambiguity detection – are there leafs with multiple valid interpretations (e.g., a bandwidth range without a selected value)?

A dry‑run validation step before invoking the renderer can catch many of these issues offline.

Example Diagnostic Tests for Intent Validation

CLI Commands for Intent Verification

If the intent resides in a NETCONF datastore, use yangcli or netconf-console:

# Validate intent instance against the operator‑intent YANG module
yangcli -s operator-intent@2023-09-01.yang \
        --datastore candidate \
        --validate-only \
        --filter /operator-intent:service[uuid='svc-1234']

A non‑zero exit code indicates a schema violation. To list missing mandatory leaves:

# List any leafs with status 'mandatory' that are absent
yangcli -s operator-intent@2023-09-01.yang \
        --datastore candidate \
        --get /operator-intent:service[uuid='svc-1234'] \
        | grep -E '^.*:.*$' | while read leaf; do
    if ! yangcli -s operator-intent@2023-09-01.yang \
                 --datastore candidate \
                 --get "/operator-intent:service[uuid='svc-1234']/$leaf" 2>/dev/null; then
        echo "Missing mandatory leaf: $leaf"
    fi
done

Code Snippet for Automated Intent Analysis

A Python script using pyangbind performs semantic checks beyond schema validation:

#!/usr/bin/env python3
from pyangbind.lib.pybindJSON import load
from pyangbind.lib.serialise import pybindJSONDecoder
import sys

def validate_intent(json_path, yang_module):
    # Load the YANG model
    module = pyangbindJSONDecoder.load_module(yang_module)
    # Load the intent instance
    try:
        intent = load(json_path, module)
    except Exception as e:
        print(f"Schema validation failed: {e}")
        return False

    # Example semantic rule: bandwidth must be a multiple of 100 Mbps
    for svc in intent.operator_intent.service:
        if svc.bandwidth_mbps % 100 != 0:
            print(f"Service {svc.uuid}: bandwidth {svc.bandwidth_mbps}Mbps not a multiple of 100Mbps")
            return False
        # Example: critical services must define a protection path
        if svc.critical and not svc.protection_path:
            print(f"Critical service {svc.uuid} lacks protection_path")
            return False
    return True

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("Usage: validate_intent.py <intent.json> <operator-intent.yang>")
        sys.exit(1)
    sys.exit(0 if validate_intent(sys.argv[1], sys.argv[2]) else 1)

Assumptions – the intent model is versioned and stored in a datastore that supports validation (NETCONF/RESTCONF); the YANG module is available locally.
Limitation – semantic rules are hard‑coded; scaling to many intent types calls for a rule engine (e.g., OPA) rather than ad‑hoc Python.

Diagnosing Stale Topology Facts

Understanding Topology Fact Collection

Topology facts are gathered by agents (LLDP, CDP, NETCONF, gNMI) and stored in a fact store (graph DB, Redis, KV store). The renderer queries this store for:

• Device identifiers and roles
• Interface names, speeds, operational state
• Neighbor relationships (loop‑free topology)
• Capabilities (supported YANG models, ACL types)

If the fact store lags behind the actual network, the renderer may reference non‑existent interfaces or incorrect neighbor IDs.

Identifying Stale Fact Indicators

• Timestamp mismatch – last_updated older than the most recent network change event (syslog, SNMP traps, telemetry).
• Missing expected entries – a device present in the inventory system absent from the fact store.
• Inconsistent degree – LLDP reports N neighbors but the fact store shows M ≠ N.

Example CLI Commands for Topology Fact Refresh

Assuming a fact store exposed via a REST API (/api/facts) and a collector script collect_facts.py:

# Force a full refresh of all facts
curl -X POST http://factstore.example.com/api/facts/refresh

# Verify the timestamp of a specific device
curl -s http://factstore.example.com/api/facts/device/router01 | jq '.last_updated'

If the collector runs as a systemd service:

sudo systemctl restart topology-collector
sudo journalctl -u topology-collector -f   # follow logs for errors

Code Example for Automated Fact Validation

A simple script compares the fact store against an authoritative inventory (e.g., NetBox) and flags stale entries:

#!/usr/bin/env python3
import requests, json, sys
from datetime import datetime, timezone, timedelta

FACT_STORE = "http://factstore.example.com/api/facts"
INVENTORY  = "http://netbox.example.com/api/dcim/devices/"
STALE_THRESHOLD = timedelta(minutes=5)   # consider stale if older than this

def fetch_facts():
    r = requests.get(FACT_STORE)
    r.raise_for_status()
    return r.json()   # expects list of dicts with keys: device_id, last_updated, interfaces, ...

def fetch_inventory():
    r = requests.get(INVENTORY)
    r.raise_for_status()
    return r.json()['results']

def is_stale(fact, inv_dev):
    fact_time = datetime.fromisoformat(fact['last_updated']).replace(tzinfo=timezone.utc)
    inv_time  = datetime.fromisoformat(inv_dev['last_updated']).replace(tzinfo=timezone.utc)
    return (datetime.now(timezone.utc) - fact_time) > STALE_THRESHOLD

def main():
    facts   = {f['device_id']: f for f in fetch_facts()}
    inventory = {d['name']: d for d in fetch_inventory()}
    stale = []
    for name, inv in inventory.items():
        if name not in facts:
            print(f"MISSING fact for device {name}")
            stale.append(name)
        elif is_stale(facts[name], inv):
            print(f"STALE fact for device {name} (fact age > {STALE_THRESHOLD})")
            stale.append(name)
    if stale:
        print(f"\n{len(stale)} devices have stale or missing facts.")
        sys.exit(1)
    else:
        print("All facts are fresh.")
        sys.exit(0)

if __name__ == "__main__":
    main()

Assumptions – the fact store exposes a JSON endpoint with an ISO‑8601 last_updated field; the inventory system (NetBox) provides a similar timestamp.
Scalability note – two HTTP GETs and O(N) comparisons are fine for ≤10 k devices (<1 s). For larger scales, use paginated APIs or stream facts via a message bus (Kafka) and validate in a stream processor.

Investigating Missing Policy Context

Policy Context Requirements

Policy context includes any global or segment‑specific rules that must be rendered into the final configuration, such as:

• ACLs / firewall rules
• QoS policies (policers, shapers, queue mappings)
• Route‑maps, prefix‑lists, BGP communities
• Security profiles (802.1X, MACsec)

The renderer typically receives policy context as a separate data structure (JSON blob keyed by policy_id) or queries a policy store (OPA, Conan, custom service) during rendering.

Diagnostic Tests for Policy Context Validation

A missing policy manifests as:

• Renderer logs indicating “policy X not found”
• Generated config lacking expected ACL statements or QoS class‑maps
• Post‑deploy validation (Batfish, Cisco NSO compliance check) reporting policy violations

Example Code for Policy Context Verification

The following Python snippet queries an OPA policy store for all policies applicable to a service intent and compares them against what the renderer actually emitted:

#!/usr/bin/env python3
import requests, json, sys

OPA_URL = "http://opa.example.com/v1/data/network/policy"
RENDERED_CONFIG_PATH = sys.argv[1]   # path to rendered config (text)
SERVICE_ID = sys.argv[2]             # UUID of the service being validated

def fetch_applicable_policies(service_id):
    payload = {
        "input": {
            "service_id": service_id,
            "attributes": {}   # could include tenant, location, etc.
        }
    }
    r = requests.post(OPA_URL, json=payload)
    r.raise_for_status()
    return r.json()['result']   # list of policy objects

def extract_policy_ids_from_config(config_text):
    # Very naive example: look for lines that start with 'policy-id:'
    ids = set()
    for line in config_text.splitlines():
        if line.startswith("policy-id:"):
            ids.add(line.split(":")[1].strip())
    return ids

def main():
    if len(sys.argv) != 3:
        print(f"Usage: {sys.argv[0]} <rendered-config> <service-id>")
        sys.exit(1)

    config_text = open(RENDERED_CONFIG_PATH).read()
    rendered_ids = extract_policy_ids_from_config(config_text)
    opa_ids = {p['id'] for p in fetch_applicable_policies(SERVICE_ID)}

    missing = opa_ids - rendered_ids
    extra   = rendered_ids - opa_ids

    if missing:
        print(f"MISSING policies in rendered config: {missing}")
    if extra:
        print(f"EXTRA policies in rendered config (possible over‑render): {extra}")

    sys.exit(0 if not missing else 1)

if __name__ == "__main__":
    main()

Assumptions – OPA returns a JSON list where each element has an id field matching the token used in the renderer’s Jinja2 template ({{ policy.id }}). The rendered config uses a simple policy-id: marker for extraction; real‑world verification would use a proper parser (e.g., ciscoconfparse for IOS, pybatfish for vendor‑agnostic checks).

CLI Commands for Policy Context Analysis

If the policy store is exposed via a REST endpoint, you can quickly check coverage:

# List all policies for tenant 'acme'
curl -s http://policy.example.com/api/policies?tenant=acme | jq '.[].id'

# Verify that a specific ACL is present in the rendered config
grep -i "access-list 101" rendered.cfg || echo "ACL 101 missing"

Identifying Bad Renderer Issues

Renderer Functionality Overview

A renderer takes three inputs:

1. Intent – high‑level service description.
2. Topology facts – current network state.
3. Policy context – applicable rules.

It then applies a templating or transformation engine (Jinja2, Go text/template, DSL‑based transpiler) to produce device‑specific configuration (CLI, YANG, JSON, or NETCONF edit‑config).

A “bad renderer” can manifest as:

• Incorrect variable substitution (e.g., using {{ interface.name }} where the fact store provides ifName).
• Logic errors in loops or conditionals (producing duplicate or missing stanzas).
• Improper escaping leading to malformed CLI (e.g., unquoted strings with spaces).
• Use of outdated templates that do not match the target OS version.

Diagnostic Tests for Renderer Validation

The most direct test is to render a known good input set and compare the output against a trusted baseline (golden config). Any deviation points to the renderer.

Example Code Snippets for Renderer Testing

A pytest‑based test suite renders a set of intents and validates against golden files:

# test_renderer.py
import os, json, jinja2, difflib
import pytest

TEMPLATE_DIR = os.path.join(os.path.dirname(__file__), "templates")
GOLDEN_DIR   = os.path.join(os.path.dirname(__file__), "golden")
INTENT_DIR   = os.path.join(os.path.dirname(__file__), "intents")
FACTS_FILE   = os.path.join(os.path.dirname(__file__), "facts.json")
POLICY_FILE  = os.path.join(os.path.dirname(__file__), "policy.json")

def load_json(path):
    with open(path) as f:
        return json.load(f)

@pytest.fixture
def env():
    return jinja2.Environment(
        loader=jinja2.FileSystemLoader(TEMPLATE_DIR),
        trim_blocks=True,
        lstrip_blocks=True,
    )

def render_intent(intent_name, env):
    intent   = load_json(os.path.join(INTENT_DIR, f"{intent_name}.json"))
    facts    = load_json(FACTS_FILE)
    policy   = load_json(POLICY_FILE)
    template = env.get_template("base.j2")
    return template.render(intent=intent, facts=facts, policy=policy)

def test_all_intents(env):
    for intent_file in os.listdir(INTENT_DIR):
        if not intent_file.endswith(".json"):
            continue
        name = os.path.splitext(intent_file)[0]
        rendered = render_intent(name, env)
        golden_path = os.path.join(GOLDEN_DIR, f"{name}.cfg")
        with open(golden_path) as f:
            golden = f.read()
        if rendered != golden:
            diff = "\n".join(difflib.unified_diff(
                golden.splitlines(),
                rendered.splitlines(),
                lineterm=''
            ))
            pytest.fail(f"Renderer output mismatch for {name}:\n{diff}")

How to use – place Jinja2 templates in templates/, intent JSON files in intents/, and the corresponding golden configs in golden/. Run pytest; any failure indicates a renderer defect.

Additional Renderer Checks

• Variable map verification – print the resolved context before rendering to ensure expected keys exist.
• Template linting – use jinja2-lint or similar to catch syntax errors.
• Version‑specific templates – maintain separate template directories per OS version and select the appropriate one based on facts.

Closing Notes

By systematically applying the tests outlined above—starting with the most observable symptom and progressing through intent validation, fact freshness, policy coverage, and renderer correctness—