The linter failures worth surfacing to operators

Introduction to AI‑Generated Configs Validation

AI‑generated network configurations often fail CI pipelines with a generic “rejected by pipeline” banner. Engineers need concrete artifacts to triage failures:

• Parser offsets – pinpoint where the syntax tree diverges.
• Unsupported knobs – list statements the target NOS does not recognize.
• Undefined variables – reveal template variables missing from the rendering context.
• Expansion diffs – show what changed after macro/variable expansion.
• Failed lab tests – expose runtime incompatibilities in a simulated topology.

Each artifact isolates a class of error, turning a black‑box rejection into an actionable checklist.

Why Detailed Validation Feedback Matters

Detailed feedback reduces mean‑time‑to‑repair (MTTR) by shifting work from guesswork to targeted fixes.

• A parser offset that highlights a missing semicolon at line 23 lets the engineer edit that line instead of scanning the whole file.
• An unsupported‑knob list showing bgp graceful-restart on a NOS that only supports bgp graceful-restart stalepath-time enables a quick replacement.
• Undefined‑variable reports stop template rendering loops before they waste compute.
• Expansion diffs expose unintended side‑effects of Jinja2 macros.
• Failed lab tests confirm that the rendered config actually brings up protocols in a simulated topology.

Together, these artifacts form a validation stack that moves beyond syntactic linting to semantic correctness.

Parser Offsets

Definition

A parser offset is the byte or character index where a configuration parser reports a syntax error. Unlike a generic “invalid config” message, the offset tells the engineer exactly where the token stream diverges from the grammar. Offsets are emitted by vendors’ CLI parsers (e.g., Cisco IOS‑XR, Juniper Junos) or by open‑source parsers such as pyang for YANG or configtree for vendor‑agnostic models. The offset enables line‑number mapping, syntax‑highlighting in editors, and automated patch generation.

Typical Use Cases

• Missing delimiter – An absent ! in an IOS‑style ACL causes the parser to consume the next line as part of the previous entry; the offset points to the start of that line, revealing the delimiter gap.
• Extra braces – In Juniper, an extra set of braces at [edit protocols bgp group] yields an offset inside the braces, indicating an unexpected block.
• YANG leaf‑ref error – When AI generates a YANG‑based config with a leaf‑ref that points to a non‑existent node, the YANG compiler reports an offset at the leaf‑ref statement, guiding the engineer to correct the path.

Example: Extracting Offsets with ciscoconfparse

from ciscoconfparse import CiscoConfParse
import re
import sys

def find_parse_offset(cfg_path):
    try:
        CiscoConfParse(cfg_path, syntax='ios')
        return None  # No error found
    except Exception as e:
        msg = str(e)
        m = re.search(r'line (\d+):', msg)
        if m:
            line_no = int(m.group(1))
            with open(cfg_path, 'r', encoding='utf-8') as f:
                data = f.read()
                lines = data.splitlines(True)
                offset = sum(len(l) for l in lines[:line_no-1])
                return offset, line_no, msg
        return None, None, msg

if __name__ == '__main__':
    offset, line, err = find_parse_offset('generated.cfg')
    if offset is None:
        print('Config parsed successfully')
    else:
        print(f'Parse error at byte offset ~{offset} (line {line}): {err}')

Notice:

• The function approximates the byte offset from line numbers; production tools (e.g., iosxr-parser) provide exact offsets.
• The exception message includes the offending token for a quick fix.
• If offset is None, the config passed the parser’s syntactic check.

Unsupported Knobs

Identification

Unsupported knobs are configuration statements that the target network operating system (NOS) does not recognize. AI models trained on mixed‑vendor corpora may emit Juniper‑style set protocols mpls label-switched-path on an IOS‑XR device, or vice‑versa. Detection relies on comparing the rendered config against the NOS feature model (often a YANG schema or vendor‑specific CLI grammar). Tools such as Batfish’s question.parse_config() or custom ansible-lint rules can flag unknown tokens.

Troubleshooting Workflow

1. Remove the knob if it is non‑essential.
2. Replace it with an equivalent supported knob (e.g., bgp graceful-restart → bgp graceful-restart stalepath-time).
3. Conditionally exclude it via Jinja2 templating based on an inventory variable that defines the NOS family.

Steps:

• Run the validation step that outputs a list of unsupported statements.
• Examine each statement’s context (parent hierarchy, indentation).
• Consult the NOS command reference to verify support.
• Adjust the template or inventory and re‑run validation.

CLI Example: Detecting Unsupported Knobs with Batfish

# Assume a Batfish service is running and a snapshot named 'lab' exists
bf session set snapshot lab

# Parse the config and collect unsupported lines
bf question.parse_config \
    --nodes 'all' \
    --output-format json \
| jq '.[] | select(.unsupported == true) | .line, .text'

Sample output (annotated):

{
  "line": 58,
  "text": "set protocols mpls label-switched-path LSP1 to 10.0.0.2",
  "unsupported": true,
  "reason": "Command not found in IOS‑XR grammar"
}

Notice:

• The line field gives the exact line number for quick editing.
• The reason field explains why the token is unsupported (missing in grammar).
• An empty output indicates all knobs are recognized by the target NOS.

Undefined Variables

Causes and Consequences

Undefined variables arise when a Jinja2 (or similar) template references a variable not supplied in the rendering context. Common causes:

• Typo in variable name ({{ bgp_as }} vs {{ bgp_asn }}).
• Missing inventory entry for a host group.
• Conditional blocks that skip variable assignment.

Consequences include template rendering aborts, empty strings that produce invalid CLI (e.g., neighbor remote-as), or silent misconfiguration if the variable defaults to an empty string and the resulting command is accepted but nonsensical.

Handling Strategies

• Fail fast: Configure Jinja2 to raise UndefinedError on missing variables (Environment(undefined=StrictUndefined)).
• Default values: Provide sensible defaults via {{ var \| default('fallback') }} only when semantics allow it.
• Pre‑render lint: Use ansible-lint or jinja2-lint to scan templates for undefined names before rendering.
• Inventory validation: Cross‑check required variables against inventory sources (NetBox, Nautobot) using a CI step that fails if any host lacks a mandatory variable.

Example: Rendering with Strict Undefined Handling

from jinja2 import Environment, StrictUndefined, FileSystemLoader
import os

def render_template(template_path, context):
    env = Environment(
        loader=FileSystemLoader(searchpath=os.path.dirname(template_path)),
        undefined=StrictUndefined,
        trim_blocks=True,
        lstrip_blocks=True,
    )
    template = env.get_template(os.path.basename(template_path))
    try:
        rendered = template.render(**context)
        return rendered, None
    except Exception as e:
        return None, str(e)

if __name__ == '__main__':
    tmpl = 'templates/bgp.j2'
    ctx = {
        'local_as': 65001,
        # 'peer_as' intentionally omitted to trigger error
    }
    rendered, err = render_template(tmpl, ctx)
    if err:
        print(f'Undefined variable error: {err}')
        # Expected output: "UndefinedError: 'peer_as' is undefined"
    else:
        print(rendered)

Notice:

• The error message contains the exact missing variable name (peer_as).
• StrictUndefined ensures the render fails rather than silently inserting an empty string.
• In a CI pipeline, a non‑zero exit code from this script blocks promotion of the config.

Expansion Diffs

What They Show

Expansion diffs display the difference between a template (with placeholders, loops, conditionals) and its fully expanded rendering after variable substitution and macro expansion. Unlike a plain diff between two static files, expansion diffs highlight what the AI‑generated template actually produces, making it easy to spot unintended side‑effects such as duplicated statements, over‑broad allow-service clauses, or missing no negations.

Review Process

1. Render the template with the intended context → expanded.cfg.
2. Generate a diff between the template file and expanded.cfg (e.g., diff -u or git diff --no-index).
3. Examine added lines: these are the concrete configuration bits that will be pushed to devices.
4. Look for patterns:
   • Blocks that appear many times → possible over‑generation.
   • Missing no prefixes on features that should be disabled.
   • Hard‑coded values that should be parameterized.
     If the diff reveals surprises, adjust the template or context and re‑render.

Example: Misplaced Command Outside a Loop

Template (interface.j2):

interface GigabitEthernet0/0
{% for vlan in vlans %}
  GigabitEthernet0/0.{{ vlan }}
{% endfor %}
 encapsulation dot1q  {{ vlan }}   <-- BUG: vlan undefined here

Context: {"vlans": [10,20,30]}

Rendered config (expanded.cfg):

interface GigabitEthernet0/0
  GigabitEthernet0/0.10
  GigabitEthernet0/0.20
  GigabitEthernet0/0.30
 encapsulation dot1q  10

Diff (diff -u interface.j2 expanded.cfg):

--- interface.j2
+++ expanded.cfg
@@
 interface GigabitEthernet0/0
+  GigabitEthernet0/0.10
+  GigabitEthernet0/0.20
+  GigabitEthernet0/0.30
+ encapsulation dot1q  10

Notice:

• The added lines show the three sub‑interfaces correctly indented.
• The stray encapsulation dot1q 10 line appears at the global scope, using the first VLAN value from the loop—a clear bug.
• The diff makes the misplaced command obvious; moving the encapsulation line inside the loop resolves it.

Failed Lab Tests

Role of Lab Tests

Lab tests instantiate a simulated topology (using Containerlab, vrnetlab, GNS3, etc.) and apply the rendered config to virtual routers. They then run protocol‑level checks (e.g., BGP session state, OSPF adjacency, ping sweep) to verify that the config achieves the intended intent. A failed lab test indicates that the config, while syntactically valid, does not produce the expected operational state.

Analyzing a Failure

Collect:

• Test logs – show which step failed (e.g., bgp peer not established).
• Router state – show bgp summary, show ip ospf neighbor, etc., captured via the test harness.
• Packet captures – if available, reveal whether packets are being sent/received or dropped.
• Configuration diff – compare the pushed config against the intended config to ensure no corruption occurred during transfer.

Proceed from the symptom (e.g., no BGP peers) to the root cause (e.g., missing update-source or incorrect AS number), adjust the template or variables, and re‑run the validation stack.

By leveraging parser offsets, unsupported‑knob lists, undefined‑variable reports, expansion diffs, and failed lab test results, engineers can move from a vague “rejected by pipeline” banner to a precise, actionable debugging workflow. This reduces MTTR and increases confidence that AI‑generated configs are both syntactically correct and semantically sound.