Introduction to Normalization Layer

The normalization layer is a critical component in network management and monitoring systems, responsible for transforming raw data from various sources into a standardized and durable format. This layer plays a crucial role in enabling efficient data analysis, drift detection, and historical comparison.

Designing the Normalization Layer

Data Ingestion and Processing

The normalization layer ingests raw data from various sources, including show interface, neighbor, and inventory outputs.

Handling Show Interface Output

The show interface output provides detailed information about network interfaces. The normalization layer processes this output by extracting relevant fields and creating a standardized data structure.

Handling Neighbor Output

The neighbor output provides information about neighboring devices. The normalization layer processes this output by extracting relevant fields and creating a standardized data structure.

Handling Inventory Output

The inventory output provides information about device inventory. The normalization layer processes this output by extracting relevant fields and creating a standardized data structure.

Data Transformation and Mapping

The normalization layer transforms the processed data into a standardized format, consisting of durable entity IDs and field-level state.

Creating Durable Entity IDs

The normalization layer creates durable entity IDs by combining relevant fields from the processed data.

Generating Field-Level State

The normalization layer generates field-level state by extracting specific attributes and values associated with each entity.

Data Modeling and Schema

The normalization layer uses a standardized data model and schema to store and manage the transformed data.

Entity ID Structure and Format

The entity ID structure and format provide a unique and persistent identifier for each entity.

Field-Level State Data Model

The field-level state data model captures specific attributes and values associated with each entity.

Data Storage and Retrieval Considerations

The normalization layer stores the transformed data in a scalable and efficient data storage system.

Implementation and Code Examples

The normalization layer is implemented using a combination of programming languages and technologies.

CLI Examples for Data Ingestion and Processing

# Ingest show interface output
$ show_interface_output=$(show interface)
# Process show interface output
$ processed_data=$(python process_show_interface.py $show_interface_output)

Using Python for Data Transformation and Mapping

import json

# Define the entity ID structure and format
def create_entity_id(device_name, interface_name, ip_address):
    return f"{device_name}-{interface_name}-{ip_address}"

# Define the field-level state data model
def generate_field_level_state(interface_operational_status, protocol_specific_details):
    return {
        "interface_operational_status": interface_operational_status,
        "protocol_specific_details": protocol_specific_details
    }

# Transform and map the data
def transform_and_map_data(processed_data):
    entity_id = create_entity_id(processed_data["device_name"], processed_data["interface_name"], processed_data["ip_address"])
    field_level_state = generate_field_level_state(processed_data["interface_operational_status"], processed_data["protocol_specific_details"])
    return {
        "entity_id": entity_id,
        "field_level_state": field_level_state
    }

Using SQL for Data Storage and Retrieval

-- Create the entity ID table
CREATE TABLE entity_ids (
    entity_id VARCHAR(255) PRIMARY KEY,
    device_name VARCHAR(255),
    interface_name VARCHAR(255),
    ip_address VARCHAR(255)
);

-- Create the field-level state table
CREATE TABLE field_level_state (
    entity_id VARCHAR(255) PRIMARY KEY,
    interface_operational_status VARCHAR(255),
    protocol_specific_details VARCHAR(255)
);

-- Insert data into the entity ID table
INSERT INTO entity_ids (entity_id, device_name, interface_name, ip_address)
VALUES ('device1-interface1-192.168.1.1', 'device1', 'interface1', '192.168.1.1');

-- Insert data into the field-level state table
INSERT INTO field_level_state (entity_id, interface_operational_status, protocol_specific_details)
VALUES ('device1-interface1-192.168.1.1', 'up', 'ospf');

Troubleshooting and Debugging

The normalization layer provides various troubleshooting and debugging tools.

Common Issues and Error Messages

Duplicate or missing entity IDs
Field-level state inconsistencies
Data storage and retrieval errors

Scaling and Performance Considerations

The normalization layer is designed to handle large volumes of data and provide high-performance querying and retrieval capabilities.

Limitations of the Normalization Layer

Handling large volumes of data
Optimizing data processing and storage

Drift Detection and Historical Comparison

The normalization layer provides various tools and techniques for drift detection and historical comparison.

Using the Normalization Layer for Drift Detection

The normalization layer can be used for drift detection by analyzing changes in entity IDs and field-level state over time.

Joins and Data Integration

The normalization layer provides various tools and techniques for joins and data integration.

Joining Normalized Data with Other Data Sources

The normalization layer can be used to join normalized data with other data sources.

Security and Access Control

The normalization layer provides various tools and techniques for security and access control.

Securing the Normalization Layer

The normalization layer can be secured by using a combination of authentication and authorization mechanisms.

From raw CLI to stable entity graphs