CLI, Scripts, and LLMs for Bring-Up Triage

Deterministic Shell Inspection

Overview

Deterministic shell inspection executes a fixed set of commands on a target system and interprets the output with deterministic rules. Unlike heuristic or ML‑based methods, the same system state always yields identical results. Common techniques include:

• Command‑line utilities (grep, awk, sed, netstat, ip, device show commands) and custom parsers for JSON/XML/YAML.
• Expect‑style automation that sends a command, waits for a specific prompt, then continues.
• Idempotent checks that return a Boolean for conditions such as “interface eth0 is up and has IP 10.0.0.1/24”.
• Artifact collection that saves raw command output for offline analysis.

Benefits

• Reproducibility: Identical hardware/software produces the same findings.
• Auditability: Every step is a visible shell command.
• Low overhead: Relies only on the target’s native toolchain.
• Predictable failure modes: Errors are explicit (non‑zero exit code, missing output).
• Compliance friendliness: Provides documented, repeatable evidence for audits.

Scripted Diffs for Lab Bring‑Up

Implementation

Scripted diffs compare a known‑good reference state with the current state of a lab device.

1. Capture reference – Run deterministic inspection on a verified node; store output as baseline.txt.
2. Collect current state – Run the same inspection on the node under test; store output as current.txt.
3. Generate diff – Use diff, vimdiff, or structured tools (jq, yq) to create a human‑readable patch.
4. Interpret diff – Flag any added/removed/changed line matching a significant pattern (e.g., missing VLAN, wrong ACL); ignore irrelevant changes (timestamps, counters).
5. Report – Emit PASS/FAIL and optionally a detailed diff for review.

Design points

• Idempotent collection – Inspection must not alter device state.
• Normalization – Strip volatile fields (timestamps, counters, dynamic IDs) before diffing.
• Structured vs. text diff – Prefer JSON/YAML for semantic diffing; otherwise use line‑based diff with ignore patterns.

Example: Bash Script for Linux Bridge Inspection

#!/usr/bin/env bash
set -euo pipefail

# ----- Configuration -----
TARGET_HOST="lab-switch01"
SSH_USER="admin"
BASELINE_DIR="/opt/lab-baselines"
CURRENT_DIR="/opt/lab-current"
INSPECTION_SCRIPT="/opt/lab-inspect/bridge_state.sh"
DIFF_TOOL="diff -u"
IGNORE_PATTERNS="(^#|^$|uptime|timestamp)"   # regex for lines to strip

# ----- Helper functions -----
run_inspection() {
    local host="$1"; local out_file="$2"
    ssh -o BatchMode=yes "${SSH_USER}@${host}" "bash -s" < "${INSPECTION_SCRIPT}" > "${out_file}"
}

normalize() {
    local in_file="$1"; local out_file="$2"
    grep -Ev "${IGNORE_PATTERNS}" "${in_file}" | sed 's/[[:space:]]*$//' | sort > "${out_file}"
}

# ----- Main workflow -----
mkdir -p "${BASELINE_DIR}" "${CURRENT_DIR}"
BASELINE_FILE="${BASELINE_DIR}/${TARGET_HOST}_bridge.txt"
CURRENT_FILE="${CURRENT_DIR}/${TARGET_HOST}_bridge.txt"
NORMALIZED_BASELINE="${BASELINE_DIR}/${TARGET_HOST}_bridge.norm.txt"
NORMALIZED_CURRENT="${CURRENT_DIR}/${TARGET_HOST}_bridge.norm.txt"

# 1. Pull baseline if not present (first‑run bootstrap)
if [[ ! -f "${BASELINE_FILE}" ]]; then
    echo "[INFO] Baseline missing; capturing from ${TARGET_HOST} as reference."
    run_inspection "${TARGET_HOST}" "${BASELINE_FILE}"
    normalize "${BASELINE_FILE}" "${NORMALIZED_BASELINE}"
    echo "[INFO] Baseline saved."
else
    normalize "${BASELINE_FILE}" "${NORMALIZED_BASELINE}"
fi

# 2. Capture current state
run_inspection "${TARGET_HOST}" "${CURRENT_FILE}"
normalize "${CURRENT_FILE}" "${NORMALIZED_CURRENT}"

# 3. Diff
if diff_output=$(${DIFF_TOOL} "${NORMALIZED_BASELINE}" "${NORMALIZED_CURRENT}" 2>&1); then
    echo "[PASS] No significant differences detected."
else
    echo "[FAIL] Differences found:"
    echo "${diff_output}"
    echo "${diff_output}" > "${CURRENT_DIR}/${TARGET_HOST}_bridge.diff"
fi

Explanation

• run_inspection executes bridge_state.sh over SSH to obtain a deterministic view of bridge interfaces, STP state, and VLAN mappings.
• normalize removes volatile lines (comments, empty lines, uptime, timestamps) and sorts the remainder to eliminate ordering noise.
• Any non‑empty diff is treated as a failure; production use may whitelist acceptable changes (e.g., learned MAC addresses).

Troubleshooting

LLM‑Powered Artifact Bundle Analysis

LLM Overview

A Large Language Model (LLM) ingests arbitrary text (or structured data serialized to text) and produces natural‑language insights, summaries, or remediation suggestions. We assume an LLM that:

• Accepts prompts of several thousand tokens.
• Can be instructed to output JSON or markdown for downstream parsing.
• Runs locally or air‑gapped (e.g., via Ollama, Llama.cpp, or a private API) to avoid data exfiltration.
• Lacks autonomous tool‑calling unless explicitly integrated; it acts as a passive analyst.

Assumptions

• Model knowledge cutoff precedes the lab’s software versions; it relies entirely on the provided artifact for factual correctness.
• Model may hallucinate; therefore output must be validated against the raw artifact before trust.
• Model does not modify the target system; it only reads the bundle.

Integration Pattern

1. Bundle creation – Run the deterministic inspection script (same as used for scripted diffs) and capture all raw output into a directory or tarball (artifact-bundle/).
2. Optional summarization – Preprocess with jq, grep, etc., to reduce token count while preserving salient data.
3. Prompt construction – Combine a static system prompt (role, output format, safety constraints) with the artifact content (inline or via file:// reference if supported).
4. LLM inference – Send the prompt to the model, collect the response.
5. Post‑processing – Parse the model’s output (expected JSON with status, findings[], recommended_actions[]) and feed it into a decision engine or ticketing system.
6. Human‑in‑the‑loop – Present the LLM’s summary to an operator for approval before any remediation.

Design considerations

• Token budget – Truncate non‑essential sections or use a retrieval‑augmented approach (e.g., grep‑based snippets) for large outputs like show tech-support.
• Deterministic prompting – Use a fixed prompt template and temperature 0.0 for repeatability.
• Safety guardrails – Instruct the model to never suggest configuration‑modifying commands without explicit human approval; embed this in the system prompt.

Code Example: Python LLM Analyzer

#!/usr/bin/env python3
import json
import os
import subprocess
import sys
from pathlib import Path

# ----- Configuration -----
OLLAMA_HOST = os.getenv("OLLAMA_HOST", "http://localhost:11434")
MODEL_NAME = "llama3:8b-instruct"   # adjust to your deployed model
ARTIFACT_DIR = Path("/opt/lab-artifacts/current")
PROMPT_TEMPLATE = """You are a senior network engineer tasked with reviewing a lab bring‑up artifact bundle.
Your goal is to identify any deviations from the expected baseline, explain the likely impact,
and propose *only* verification steps (no configuration changes) that a human operator should perform.
Respond in valid JSON with the following schema:
{
  "status": "PASS" | "FAIL" | "UNCERTAIN",
  "findings": [
    { "description": string, "severity": "low"|"medium"|"high", "evidence": string }
  ],
  "recommended_actions": [
    { "action": string, "rationale": string }
  ]
}
Do not include any extra text outside the JSON object.
ARTIFACT_BUNDLE_START
{artifact}
ARTIFACT_BUNDLE_END
"""

def build_prompt(artifact_dir: Path) -> str:
    sections = []
    for file_path in sorted(artifact_dir.rglob("*")):
        if file_path.is_file():
            try:
                content = file_path.read_text(errors="replace")
            except Exception as e:
                content = f"<<ERROR READING FILE: {e}>>"
            sections.append(f"=== FILE: {file_path.relative_to(artifact_dir)} ===\n{content}")
    artifact_text = "\n\n".join(sections)
    return PROMPT_TEMPLATE.format(artifact=artifact_text)

def query_ollama(prompt: str) -> dict:
    cmd = [
        "curl", "-s", "-X", "POST", f"{OLLAMA_HOST}/api/generate",
        "-d", json.dumps({
            "model": MODEL_NAME,
            "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0.0}
        })
    ]
    result = subprocess.run(cmd, capture_output=True, text=True, check=True)
    resp_json = json.loads(result.stdout)
    raw_text = resp_json.get("response", "").strip()
    if raw_text.startswith("```"):
        raw_text = raw_text.strip("`")
        if raw_text.startswith("json"):
            raw_text = raw_text[4:].lstrip()
    try:
        return json.loads(raw_text)
    except json.JSONDecodeError as e:
        print(f"ERROR: Failed to parse LLM output as JSON: {e}", file=sys.stderr)
        print(f"LLM raw output:\n{raw_text}", file=sys.stderr)
        sys.exit(1)

def main():
    if not ARTIFACT_DIR.is_dir():
        print(f"ERROR: Artifact directory not found: {ARTIFACT_DIR}", file=sys.stderr)
        sys.exit(1)
    prompt = build_prompt(ARTIFACT_DIR)
    print("[INFO] Sending prompt to LLM...", file=sys.stderr)
    try:
        analysis = query_ollama(prompt)
    except subprocess.CalledProcessError as e:
        print(f"ERROR: Ollama request failed: {e}", file=sys.stderr)
        sys.exit(1)
    if "status" not in analysis or "findings" not in analysis:
        print("ERROR: LLM response missing required fields", file=sys.stderr)
        sys.exit(1)
    print(json.dumps(analysis, indent=2))

if __name__ == "__main__":
    main()

Explanation

• The script builds a prompt that clearly delimits the artifact bundle.
• Temperature is set to 0.0 to reduce stochastic variation.
• The model is instructed to output only JSON; any surrounding markdown fences are stripped.
• On invalid JSON, the script exits with error, forcing the operator to inspect the raw LLM output.

CLI Example: Running LLM Analysis

# 1. Generate artifact bundle (same inspection as used for diffs)
mkdir -p /opt/lab-artifacts/current
ssh admin@lab-switch01 "bash -s" < /opt/lab-inspect/bridge_state.sh > /opt/lab-artifacts/current/bridge_state.txt
# Optionally add logs, config files, etc.
cp /var/log/syslog /opt/lab-artifacts/current/syslog.txt
cp /etc/network/interfaces /opt/lab-artifacts/current/interfaces.txt

# 2. Run LLM analysis
python3 /opt/lab-tools/llm_analyze.py