Design an LLM‑Assisted Workflow that Knows When One Command Is Insufficient, Requests Follow‑up CLI, and Merges Interface, LLDP, ARP, and Inventory Outputs into a Single Normalized Incident View Without Fabricating Missing Links

Author: Sophia Lin – Cloud & Automation Architect

Introduction to LLM‑Assisted Workflow

Overview of LLM Technology

Large Language Models (LLMs) are statistical sequence models trained on vast text corpora to predict the next token given a prompt. Modern instruction‑tuned variants (e.g., Llama‑2‑Chat, Mistral‑Instruct, or proprietary APIs) can follow natural‑language instructions, invoke external tools via function calling or ReAct patterns, and generate structured outputs when constrained by a schema. Their core capability relevant to network operations is language understanding, not factual knowledge; they excel at translating intent into a sequence of actions but do not inherently possess up‑to‑date device state.

Benefits of LLM‑Assisted Workflow

When correctly bounded, an LLM can:

1. Reduce cognitive load by translating a high‑level symptom (“users cannot reach VLAN 10”) into a deterministic investigation plan.
2. Adaptively select commands based on intermediate results, avoiding a static, over‑collecting script.
3. Provide a natural‑language narrative that couples raw CLI output with operator‑friendly reasoning, useful for handoffs and post‑mortems.
4. Enable tool‑call orchestration (e.g., via LangChain, LlamaIndex, or custom agents) where the LLM decides when to run another CLI command.

These benefits are only realizable if the workflow enforces strict boundaries on hallucination, validates every piece of data against a trusted source, and provides explicit rollback or escalation paths.

Designing the LLM‑Assisted Workflow

Identifying Insufficient Commands

The LLM does not “know” when data is missing; instead, we encode insufficiency criteria as deterministic checks that the LLM can reference in its reasoning loop:

The LLM receives these criteria as part of the system prompt (or as a retrieval‑augmented knowledge base). When its generated plan includes a command, the executor runs it, returns raw text, and a post‑processor evaluates the criteria. If any criterion fires, the post‑processor injects a follow‑up request into the LLM’s context, prompting the next iteration.

Requesting Follow‑up CLI

We adopt a tool‑calling loop similar to the ReAct pattern:

1. Prompt → LLM proposes a thought (reasoning) and an action (CLI command).

2. Executor runs the command via a secure SSH/NETCONF wrapper (e.g., scrapli or napalm).

3. Result parser converts raw CLI text into a structured intermediate representation (IR) – see Normalizing Incident View below.

4. Insufficiency checker runs against the IR; if any check fails, it appends an observation message to the conversation:

   Observation: The command 'show interfaces' did not return LLDP neighbor data for GigabitEthernet0/1.
   Please run 'show lldp neighbors' to obtain missing link information.

5. LLM receives the updated context, generates a new thought/action, and the loop repeats until all insufficiency checks pass or a maximum iteration limit (e.g., 5) is reached.

Key design choices

• Tool definition: Each CLI command is exposed as a JSON‑schema‑typed function (run_cli(command: str) -> str). The LLM only sees the schema, not the raw implementation, preventing arbitrary command injection.
• Sandboxing: The executor runs in a restricted Linux namespace with only the needed SSH key and no ability to spawn shells beyond the target device.
• Timeout & retry: Each CLI call has a hard timeout (e.g., 10 s); on failure the LLM receives an Observation: Command timed out and may decide to escalate or abort.

Merging Interface, LLDP, ARP, and Inventory Outputs

After the loop terminates, we possess four (or more) IR blobs:

• Interfaces: list of {name, admin_state, oper_state, speed, duplex, mac_address, description}
• LLDP neighbors: list of {local_port, neighbor_device_id, neighbor_port, neighbor_capabilities}
• ARP table: list of {ip_address, mac_address, interface, age}
• Inventory: list of {chassis_id, slot, part_number, serial_number, firmware_version}

A normalizer merges these blobs into a single canonical incident view by performing inner joins on stable keys (mac_address, interface name, chassis_id). The result is a JSON object that conforms to a pre‑defined schema (see Data Normalization Techniques). Missing fields are explicitly set to null and flagged with a data_quality attribute (complete, partial, missing).

Normalizing Incident View

Handling Missing Links

Missing links appear when:

• An interface is operationally up but has no LLDP neighbor (neighbor_device_id: null).
• An ARP entry lacks a corresponding MAC‑table entry (mac_address: null in the MAC table).
• Inventory does not contain a chassis entry for a detected slot (slot_data: null).

The normalizer records each missing link as a distinct object with:

{
  "link_type": "lldp_neighbor",
  "local_interface": "GigabitEthernet0/1",
  "missing_field": "neighbor_device_id",
  "confidence": 0.95,
  "source": ["show interfaces", "show lldp neighbors"]
}

The confidence score is derived from the proportion of expected data present (e.g., if 3 of 4 expected LLDP TLVs are received, confidence = 0.75). This enables downstream operators to prioritize investigation.

Preventing Fabrication of Missing Links

To avoid hallucination we enforce three layers of validation:

1. Schema‑driven generation – The LLM is prompted to output only actions (CLI commands) and observations; the final incident view is assembled exclusively by deterministic code, not by the LLM.
2. Cross‑source corroboration – A link is considered present only if at least two independent sources agree (e.g., LLDP neighbor MAC matches ARP‑derived MAC). Single‑source claims are marked unverified.
3. Inventory whitelist – Any device ID, chassis ID, or part number that does not appear in the authoritative inventory database (CMDB) is rejected and replaced with a placeholder_unknown flag, triggering a manual review.

If the LLM attempts to fabricate a command (e.g., show fake command), the executor returns an error; the LLM receives an observation of failure and is discouraged from repeating the same pattern via a negative reward in the prompt (e.g., “Invalid command; please choose from the allowed list”).

Data Normalization Techniques

• Parsing: Use vendor‑specific parsers (e.g., textfsm, ntc-templates, or genie) to convert CLI text to a structured dictionary.
• Canonical modeling: Define a JSON Schema (draft‑07) that captures the union of all needed fields. Example snippet:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "NetworkIncidentView",
  "type": "object",
  "properties": {
    "device": {"type": "string"},
    "timestamp": {"type": "string", "format": "date-time"},
    "interfaces": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": {"type": "string"},
          "admin_state": {"type": "string"},
          "oper_state": {"type": "string"},
          "mac_address": {"type": ["string", "null"]},
          "lldp_neighbor": {
            "type": ["object", "null"],
            "properties": {
              "device_id": {"type": ["string", "null"]},
              "port_id": {"type": ["string", "null"]},
              "capabilities": {"type": ["string", "null"]}
            },
            "required": ["device_id", "port_id"]
          },
          "arp_entry": {
            "type": ["object", "null"],
            "properties": {
              "ip_address": {"type": ["string", "null"]},
              "mac_address": {"type": ["string", "null"]},
              "age": {"type": ["integer", "null"]}
            },
            "required": ["ip_address"]
          }
        },
        "required": ["name", "admin_state", "oper_state"]
      }
    },
    "inventory": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "chassis_id": {"type": "string"},
          "slot": {"type": ["integer", "null"]},
          "part_number": {"type": "string"},
          "serial_number": {"type": "string"},
          "firmware_version": {"type": "string"}
        },
        "required": ["chassis_id", "part_number", "serial_number"]
      }
    },
    "data_quality": {
      "type": "object",
      "additionalProperties": {"enum": ["complete", "partial", "missing", "unverified"]}
    }
  },
  "required": ["device", "timestamp", "interfaces", "inventory", "data_quality"]
}

• Post‑processing: After merging, run a JSON Schema validator (e.g., jsonschema Python library). Any validation error triggers a manual review ticket and prevents the view from being stored in the incident database.

Implementing the LLM‑Assisted Workflow

Code Examples for CLI Integration

Below is a minimal, production‑skeleton Python implementation using scrapli for CLI, OpenAI function calling (or any compatible LLM endpoint), and jsonschema for validation. The code assumes a local LLM endpoint that respects the OpenAI chat completions API with functions support.

# workflow.py
import json
import asyncio
from typing import List, Dict, Any
import jsonschema
from scrapli import AsyncScrapli
from scrapli.exceptions import ScrapliFailure

# ---------- CONFIG ----------
DEVICE = {
    "host": "10.0.0.1",
    "auth_username": "admin",
    "auth_password": "*****",
    "auth_strict_key": False,
    "platform": "cisco_iosxe",
}
LLM_ENDPOINT = "http://localhost:8000/v1/chat/completions"  # e.g., vLLM or TGI
MAX_ITERATIONS = 5
# ---------------------------

# Tool schema exposed to the LLM
TOOLS = [
    {
        "name": "run_cli",
        "description": "Execute a CLI command on the target device and return raw output.",
        "parameters": {
            "type": "object",
            "properties": {
                "command": {
                    "type": "string",
                    "description": "CLI command to run (e.g., 'show interfaces')."
                }
            },
            "required": ["command"],
            "additionalProperties": False
        }
    }
]

# Insufficiency checks (simplified)
def check_insufficiency(parsed: Dict[str, Any]) -> List[str]:
    msgs = []
    for iface in parsed.get("interfaces", []):
        if iface.get("oper_state") == "up" and not iface.get("lldp_neighbor"):
            msgs.append(
                f"Interface {iface['name']} is up but lacks LLDP neighbor data."
            )
        # ARP vs MAC table check omitted for brevity
    return msgs

# Normalizer (placeholder – would call textfsm/genie)
def normalize(raw_outputs: Dict[str, str]) -> Dict[str, Any]:
    # In reality: parse each raw string via appropriate parser, then merge.
    # Here we just stub a structure.
    return {
        "device": DEVICE["host"],
        "timestamp": "2025-09-26T12:00:00Z",
        "interfaces": [],  # filled by parser
        "inventory": [],   # filled by parser
        "data_quality": {}
    }

async def execute_cli(command: str) -> str:
    try:
        async with AsyncScrapli(**DEVICE) as conn:
            response = await conn.send_command(command)
            return response.result
    except ScrapliFailure as exc:
        return f"ERROR: {exc}"

This skeleton demonstrates the core loop: the LLM proposes a CLI command via the run_cli tool, the executor runs it, the output is parsed and checked for insufficiency, and the process repeats until a complete, validated incident view is produced.

End of document.