Comparing Deterministic CLI Workflows, Scripted State Collection, and an AI‑Assisted Triage Workbench for OSPF Adjacency Troubleshooting

Introduction

Three complementary approaches can be used to diagnose a stuck OSPF adjacency:

• Deterministic CLI workflows – manual, step‑by‑step command entry with immediate feedback.
• Scripted state collection – automated gathering of show‑command output into structured data for repeatable analysis.
• AI‑assisted triage workbench – a read‑only LLM‑driven interface that reasons over the collected evidence and suggests hypotheses, without making configuration changes.

All three rely on the same evidence set (the output of a defined list of OSPF‑related show commands) and the AI is constrained to read‑only tools; it may not alter device state.

Deterministic CLI Workflows

How It Works

The operator logs into the router, runs a predetermined sequence of CLI commands, interprets the output (optionally with simple grep/awk filters), and decides the next step based on observable facts. Because the same commands on the same device state produce identical output, the workflow is deterministic and fully reproducible when the command log is retained.

Advantages

• Immediate feedback – output appears as soon as the command is entered.
• No tooling dependency – works on any device with a CLI.
• Precise control – the operator chooses the next command based on the current output.
• Auditability – each keystroke can be logged via terminal history or session recording.

Disadvantages

• Time‑consuming at scale – manual login to dozens or hundreds of routers is impractical.
• Prone to human error – mistyped commands, missed output, or incorrect interpretation.
• Limited reproducibility unless the exact command sequence is recorded.
• High cognitive load – the operator must retain multiple evidence fragments while navigating the CLI.

Typical OSPF Adjacency Troubleshooting Commands

Step‑by‑Step CLI Guide

1. Establish session – SSH/console to the router where adjacency is stuck.
2. Confirm OSPF process – show ospf → note process ID and area.
3. List neighbors – show ospf neighbor → identify the neighbor not in Full state.
4. Inspect the problematic interface – show ospf interface <iface> → verify hello/dead timers match the neighbor.
5. Check IP connectivity – ping <neighbor‑ip> (if allowed) or show ip interface brief to confirm interface is up.
6. Review LSDB – show ospf database → look for the neighbor’s router‑LSA; note if missing or stale.
7. Examine physical/log errors – show interfaces <iface> → check for increasing error counters.
8. Verify filters – show access-lists applied to the interface → ensure OSPF (protocol 89) is not blocked.
9. Form hypothesis – based on mismatched timers, missing LSA, or physical errors.
10. Test hypothesis – adjust timers, clear OSPF process (clear ospf process), or fix physical layer, then re‑check neighbor state.
11. Document – capture the final show ospf neighbor output showing Full state and any configuration changes made.

Scripted State Collection

How It Works

A pre‑written script (Python/Bash, often using frameworks such as Nornir, Ansible, or PyATS) logs into devices, executes a defined command set, and stores the results in a structured format (JSON, YAML, CSV, or a time‑series database). The script removes manual typing and enables repeatable, version‑controlled data collection across many devices.

Benefits

• Repeatability – identical output format on any number of devices.
• Parallel execution – tools like Nornir or Ansible can collect data from hundreds of routers concurrently.
• Version‑controlled – scripts live in Git; changes are reviewable and auditable.
• Structured data – output can be stored as JSON/YAML for downstream analysis (e.g., with Pandas or a SIEM).
• Reduced manual effort – eliminates repetitive typing and copy‑pasting.

Limitations

• Script maintenance – CLI output formats vary across OS versions; parsers must be updated.
• Dependency on device access – requires valid credentials and reachable management plane.
• Potential for incomplete data – if a command fails or times out, the script may skip it unless error handling is explicit.
• No autonomous reasoning – the script only gathers data; interpretation still requires a human or another system.
• Risk of script‑induced load – aggressive polling can impact CPU on low‑end devices.

Example Python/Nornir Script

# collect_ospf_state.py
from nornir import InitNornir
from nornir_netmiko.tasks import netmiko_send_command
from nornir_utils.plugins.functions import print_result
import json
import os

def collect_ospf(task):
    # Gather three key show commands
    commands = [
        "show ospf neighbor",
        "show ospf interface",
        "show ospf database"
    ]
    result = {}
    for cmd in commands:
        res = task.run(task=netmiko_send_command, command_string=cmd)
        if res.failed:
            result[cmd] = {"error": str(res.exception)}
        else:
            # Store raw output; could be parsed later with textfsm or ttp
            result[cmd] = res.result
    # Create host‑specific directory
    host_dir = os.path.join("collected", task.host)
    os.makedirs(host_dir, exist_ok=True)
    out_path = os.path.join(host_dir, "ospf_state.json")
    with open(out_path, "w") as f:
        json.dump(result, f, indent=2)
    return result

def main():
    nr = InitNornir(config_file="config.yaml")  # inventory, credentials, etc.
    agg = nr.run(task=collect_ospf)
    print_result(agg)

if __name__ == "__main__":
    main()

Assumptions

• Devices are reachable via SSH/Netconf.
• netmiko_send_command returns the full CLI output as a string.
• No parsing is performed; raw output is stored for later manual or AI‑assisted review.

Integrating Scripted Collection with CLI Workflows

1. Baseline collection – Run the script nightly or on demand to build a repository of known‑good OSPF states.
2. Delta analysis – When an adjacency is reported stuck, run the script on the affected device(s) and diff the current JSON against the baseline using jq or a Python diff script.
3. Focused CLI – Use the diff output to narrow down which specific command(s) show anomalies, then jump to those commands in an interactive CLI session for deeper inspection (e.g., show ospf interface <iface> | include hello|dead).
4. Automation loop – Optionally feed the diff into a ticketing system or chat‑ops bot that presents the operator with a concise summary and suggested next CLI commands.

AI‑Assisted Triage Workbench

Architecture

+-------------------+       +-------------------+       +-------------------+
|   Operator UI     | <---->|   Orchestrator    | <---->|   LLM Engine      |
| (chat / web UI)   |       | (tool‑calling)    |       | (local or remote)|
+-------------------+       +-------------------+       +-------------------+
          ^                         ^                         ^
          |                         |                         |
          |   Evidence JSON/YAML    |   Prompt + Tools        |   Model Weights
          |   (from script)         |   (retriever, safety)   |
          +-------------------------+-------------------------+

• Operator UI – lightweight web‑based chat interface (e.g., Open WebUI, Mattermost plugin, custom React app) where the engineer types natural‑language queries.
• Orchestrator – Python/FastAPI service that:
  1. Receives the user message.
  2. Retrieves the relevant evidence set (pre‑collected JSON files) from a datastore.
  3. Constructs a prompt containing: system description (role and limits), the evidence (truncated or summarized to fit token limits), the user question, and a list of allowed tools (read‑only functions such as show_ospf_neighbor, show_ospf_interface, lookup_doc).
  4. Calls the LLM; the model may respond with a tool‑call request.
  5. Executes the tool (deterministic function that returns a specific piece of evidence or runs a read‑only CLI command via an SSH jump host).
  6. Returns the tool output to the LLM for further reasoning, iterating until the model decides to stop.
  7. Returns the final answer to the UI, clearly marking which parts are model‑generated versus evidence‑derived.
• LLM Engine – locally hosted model (e.g., Llama‑3 8B, Mistral‑7B) or a remote API with strict data‑governance guarantees. The model is not permitted to issue configuration‑change commands; its toolset is limited to read‑only queries.

Advantages

• Rapid hypothesis generation – the model can correlate multiple evidence fields faster than a human scanning logs.
• Natural‑language interaction – operators unfamiliar with CLI syntax can ask “Why is OSPF neighbor stuck on ge‑0/0/1?” and receive a concise explanation.
• Context preservation – the workbench keeps the entire conversation and evidence set in context, reducing the need to repeat commands.
• Bias toward safe actions – by restricting the toolset to read‑only operations, the AI cannot inadvertently change device state.
• Documentation surfacing – the model can pull relevant vendor documentation or RFC excerpts to support its reasoning.

Limitations & Constraints

• Hallucination risk – the model may infer facts not present in the evidence; any conclusion must be cross‑checked against the raw evidence.
• Token limits – large evidence sets (many devices, full LSDB) may exceed the context window; requires summarization or retrieval‑augmented generation (RAG).
• Tool‑call latency – each tool invocation adds network round‑trip time; deep reasoning loops can become slow.
• Dependence on script quality – if the collection script misses a command, the AI lacks that evidence.
• No guaranteed completeness – the AI may stop early if it believes it has sufficient information, potentially overlooking a subtle issue.
• Operator trust – engineers must verify AI suggestions before acting; over‑reliance can lead to missed steps.

Example AI‑Driven Diagnostic Scenarios

Scenario 1 – Mismatched Hello/Dead Intervals
Operator asks: “Why is OSPF neighbor on ge‑0/0/3 stuck in Init state?”

1. Orchestrator retrieves show ospf interface ge-0/0/3 and show ospf neighbor JSON.
2. LLM sees hello = 10s, dead = 40s locally; neighbor shows hello = 5s, dead = 20s (from show ospf neighbor detail).
3. LLM concludes: “Hello/dead timers mismatch; adjust local timers to match neighbor or vice‑versa.”
4. UI returns the conclusion with evidence snippets and a read‑only suggestion (show ospf interface ge-0/0/3 | include hello|dead). No configuration change is proposed.

Scenario 2 – OSPF Authentication Failure
Operator asks: “OSPF neighbor on te‑0/1/0 is down; what could be wrong?”

1. Evidence includes show ospf interface te-0/1/0 (shows MD5 authentication) and show ospf neighbor (state Down).
2. LLM notes that the neighbor’s interface shows no authentication configured (from show ospf interface on the neighbor).
3. LLM suggests: “Authentication mismatch; verify MD5 key chain on both ends.”
4. UI provides the evidence and a read‑only command to check the key chain (show ospf md5-key).

Scenario 3 – Physical Layer Issue
Operator asks: “OSPF neighbor on xe‑2/0/1 flaps; any clues?”

1. Evidence: show interfaces xe-2/0/1 shows increasing CRC errors and input drops.
2. LLM correlates flaps with error counters and suggests: “Likely physical layer problem; check fiber, transceiver, or port speed/duplex.”
3. UI returns the analysis and a read‑only command to monitor errors (monitor interface xe-2/0/1).

In each scenario the AI never issues a command that changes state; it only proposes hypotheses and points to specific read‑only checks.

Comparison of Diagnostic Approaches

Deterministic CLI vs. Scripted State Collection

Scripted State Collection vs. AI‑Assisted Triage Workbench

Closing Note

Each method offers a distinct trade‑off between immediacy, scalability, and interpretive power. Deterministic CLI provides the most direct, unmediated view of device state but does not scale. Scripted state collection brings repeatability and scale to data gathering, leaving analysis to the operator. The AI‑assisted triage workbench adds rapid, natural‑language reasoning over the collected evidence while remaining strictly read‑only, ensuring that the operator retains final