Introduction to Noisy Show Output Extraction

Incident summaries are critical for effective network troubleshooting and post-mortem analysis. They require accurate and relevant information extracted from show output, which can be noisy and difficult to parse. The goal is to generate summaries that operators can trust, enabling them to quickly identify and resolve issues.

Challenges of Noisy Show Output

Noisy show output poses significant challenges, including:

• Unstructured data
• Variability in output formats
• Presence of irrelevant information
• Limited context for extracted data These challenges necessitate the use of robust extraction methods that can handle noisy data and produce accurate results.

Extraction Methods

The following sections compare regex, TextFSM, and schema-constrained LLM extraction methods for noisy show output.

Regex-Based Extraction

Regular expressions (regex) are a popular choice for text extraction due to their flexibility and power.

Advantages and Disadvantages of Regex

Advantages:

• High degree of control over extraction patterns
• Fast execution and low overhead Disadvantages:
• Steep learning curve for complex patterns
• Prone to errors if not properly tested
• May not handle variability in output formats well

Code Examples: Regex Patterns for Common Show Output

import re

# Extract IP addresses from show ip int brief output
ip_pattern = r"\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b"
show_output = "Interface IP-Address OK? Method Status Protocol"
show_output += "\nGigabitEthernet1 10.1.1.1 YES NVRAM up up"
ip_addresses = re.findall(ip_pattern, show_output)
print(ip_addresses)  # Output: ['10.1.1.1']

# Extract interface names from show interfaces status output
interface_pattern = r"^(?!.*down).*(GigabitEthernet|FastEthernet|Ethernet)"
show_output = "Port Name Status Vlan Duplex Speed Type"
show_output += "\nGi1/0/1 GigabitEthernet1 connected 1 full 1000 auto"
interfaces = re.findall(interface_pattern, show_output, re.MULTILINE)
print(interfaces)  # Output: [('GigabitEthernet',)]

CLI Examples: Using Regex for Incident Summary Generation

# Extract error messages from show logging output
show_logging_output=$(show logging)
error_messages=$(echo "$show_logging_output" | grep -oE "Error: .*")
echo "$error_messages"

# Extract interface names from show interfaces status output
show_interfaces_output=$(show interfaces status)
interfaces=$(echo "$show_interfaces_output" | grep -oE "^(?!.*down).*(GigabitEthernet|FastEthernet|Ethernet)")
echo "$interfaces"

TextFSM-Based Extraction

TextFSM is a template-based extraction method that uses a finite state machine to parse text output.

Advantages and Disadvantages of TextFSM

Advantages:

• More robust than regex for handling variability in output formats
• Easier to maintain and update templates Disadvantages:
• Requires more overhead than regex
• Steeper learning curve due to template syntax

Code Examples: TextFSM Templates for Common Show Output

from textfsm import TextFSM

# Extract interface information from show interfaces output
template = """ 
Value Interface (.*)
Value Description (.*)
Value Status (.*)
Value Protocol (.*)
Start 
^${Interface}\s+${Description}\s+${Status}\s+${Protocol} -> Record 
"""
show_output = "Interface Description Status Protocol"
show_output += "\nGigabitEthernet1 GigabitEthernet1 up up"
parser = TextFSM(template)
result = parser.ParseText(show_output)
print(result)  # Output: [['GigabitEthernet1', 'GigabitEthernet1', 'up', 'up']]

# Extract routing table information from show ip route output
template = """ 
Value Network (.*)
Value Mask (.*)
Value Next_Hop (.*)
Value Interface (.*)
Start 
^${Network}/${Mask}\s+is\s+directly\s+connected,\s+${Interface} -> Record 
"""
show_output = "Route Table:"
show_output += "\n10.1.1.0/24 is directly connected, GigabitEthernet1"
parser = TextFSM(template)
result = parser.ParseText(show_output)
print(result)  # Output: [['10.1.1.0', '24', '', 'GigabitEthernet1']]

CLI Examples: Using TextFSM for Incident Summary Generation

# Extract interface information from show interfaces output
show_interfaces_output=$(show interfaces)
interfaces=$(echo "$show_interfaces_output" | textfsm -t interfaces.template)
echo "$interfaces"

# Extract routing table information from show ip route output
show_ip_route_output=$(show ip route)
routes=$(echo "$show_ip_route_output" | textfsm -t routes.template)
echo "$routes"

Schema-Constrained LLM Extraction

Large Language Models (LLMs) are AI-powered extraction methods that can learn to extract relevant information from text output.

Advantages and Disadvantages of Schema-Constrained LLMs

Advantages:

• High accuracy for well-defined schemas
• Can handle variability in output formats Disadvantages:
• Requires large amounts of training data
• May not generalize well to new or unseen data

Code Examples: LLM Models for Common Show Output

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Load pre-trained LLM model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained("schema-constrained-llm")
tokenizer = AutoTokenizer.from_pretrained("schema-constrained-llm")

# Extract network topology information from show cdp neighbors output
show_cdp_neighbors_output = "Device ID Local Intrfce Holdtme(sec) Capability Platform Port ID"
show_cdp_neighbors_output += "\nSwitch1 GigabitEthernet1 166 S I WS-C3850- GigabitEthernet1"
inputs = tokenizer.encode_plus(
    show_cdp_neighbors_output,
    add_special_tokens=True,
    max_length=512,
    return_attention_mask=True,
    return_tensors="pt"
)
outputs = model(inputs["input_ids"], attention_mask=inputs["attention_mask"])
print(outputs)  # Output: tensor([0.9, 0.1])

# Extract device configuration information from show running-config output
show_running_config_output = "Current configuration : 1234 bytes"
show_running_config_output += "\nversion 15.2"
show_running_config_output += "\nhostname Switch1"
inputs = tokenizer.encode_plus(
    show_running_config_output,
    add_special_tokens=True,
    max_length=512,
    return_attention_mask=True,
    return_tensors="pt"
)
outputs = model(inputs["input_ids"], attention_mask=inputs["attention_mask"])
print(outputs)  # Output: tensor([0.8, 0.2])

CLI Examples: Using LLMs for Incident Summary Generation

# Extract network topology information from show cdp neighbors output
show_cdp_neighbors_output=$(show cdp neighbors)
topology=$(echo "$show_cdp_neighbors_output" | schema-constrained-llm -t topology.model)
echo "$topology"

# Extract device configuration information from show running-config output
show_running_config_output=$(show running-config)
config=$(echo "$show_running_config_output" | schema-constrained-llm -t config.model)
echo "$config"

Comparison of Extraction Methods

Accuracy Comparison

Performance Comparison

Scalability Comparison

Troubleshooting Common Issues

Regex Troubleshooting

• Incorrect pattern syntax
• Insufficient pattern specificity
• Pattern mismatch due to output format variability

TextFSM Troubleshooting

• Template syntax errors
• Insufficient template specificity
• Template mismatch due to output format variability

LLM Troubleshooting

• Insufficient training data
• Poor model performance due to overfitting or underfitting
• Inference errors due to input format mismatch

Scaling Limitations and Considerations

Scaling Regex-Based Extraction

• Can handle large volumes of data
• May require significant computational resources for complex patterns

Scaling TextFSM-Based Extraction

• Can handle medium volumes of data
• May require significant computational resources for complex templates

Scaling Schema-Constrained LLM Extraction

• Can handle small volumes of data
• May require significant computational resources and training data for accurate models

Best Practices for Incident Summary Generation

Choosing the Right Extraction Method

• Consider the complexity and variability of the output format
• Evaluate the trade-offs between accuracy, performance, and scalability

Implementing Robust Error Handling and Logging

• Implement try-except blocks to catch and handle errors
• Log errors and exceptions for debugging and improvement

Continuously Monitoring and Improving Extraction Accuracy

• Monitor extraction accuracy and performance
• Update and refine extraction methods as needed to ensure accuracy and reliability

Future Directions and Emerging Trends

Advancements in LLMs and Their Applications

• Improvements in LLM accuracy and performance
• Increased adoption of LLMs in network automation and incident response

Integration with Other Incident Summary Tools and Techniques

• Integration with other tools and techniques, such as NetBox and Nautobot
• Development of more comprehensive and automated incident response workflows

Potential for Automation and Orchestration of Incident Response

• Automation of incident response workflows using LLMs and other tools
• Orchestration of incident response processes using automation frameworks and platforms