Post Mortem of a Broken Correlation Key

Introduction to Incident Narrative

Device Identifier Overview

In our multi‑tenant network automation platform each physical or virtual device is represented by a device identifier that serves as the primary key across:

• CMDB (NetBox) – device.name field, must be unique per tenant.
• Monitoring (Prometheus + SNMP exporter) – instance label derived from sysName.
• Ticketing/ServiceNow – ci_identifier field populated from the CMDB sync.
• Validation framework (Nornir + Batfish) – inventory host name pulled from NetBox.

The identifier must conform to the regex ^[a-zA-Z0-9][a-zA-Z0-9\-_.]{0,62}$ (alphanumeric start, length ≤ 63, only hyphen, underscore, dot). This guarantees safe use as a DNS label, a Kubernetes annotation, and a file‑system path without additional escaping.

Malformed Device Identifier Impact

During a routine change window a leaf switch was provisioned with the name leaf‑01‑(prod). The parentheses and trailing hyphen violated the convention, causing:

• CMDB sync failure – the import script rejected the record, leaving the device absent from NetBox.
• Monitoring mismatch – the SNMP exporter still reported sysName=leaf-01-(prod), creating an instance label that did not match any CMDB entry.
• Ticketing orphan – ServiceNow created a CI with the malformed name, but the reconciliation job could not link it to a NetBox record, producing a “stale CI” flag.
• Validation framework skip – Nornir inventory built from NetBox omitted the device, so compliance or connectivity tests were not executed on it.

The same physical device appeared in three systems under three different keys, splitting the incident narrative across tenants and delaying correlation.

Incident Review and Analysis

Initial Incident Response

The incident was treated as a single‑device BGP flap. Because the device identifier was inconsistent, post‑mortem data collection pulled logs from three separate sources, each referencing a different identifier, forcing analysts to manually correlate events.

Identification of Malformed Device Identifier

At 12:45 UTC the NetBox reconciliation script emitted:

$ ./netbox_sync.py --tenant ACME
...
ERROR: Device name 'leaf-01-(prod)' contains invalid characters (parentheses) – skipping.

A grep of the provisioning CSV revealed the offending line:

$ grep -n leaf-01-(prod) /opt/provisioning/devices.csv
42:leaf-01-(prod),acme,leaf,sw,10.0.0.15

The CSV generator had concatenated a user‑provided site code ((prod)) without sanitizing it, producing the malformed identifier.

Cross‑Tenant Incident Narrative Split

Because the identifier was malformed:

• Tenant A (the owning tenant) saw the incident only in their monitoring dashboard (Prometheus) and in the ticket created by their own ServiceNow instance.
• Tenant B (shared services) saw a “stale CI” alert in their ServiceNow instance but no corresponding monitoring alert, leading them to believe the issue was a CMDB sync problem.
• The platform team saw a NetBox sync failure but no active alerts, interpreting it as a low‑priority data‑quality issue.

Thus the same physical event generated three parallel narrative streams, each with its own timeline, causing duplicated effort and a delayed realization that the root cause was a single malformed device identifier.

Troubleshooting and Mitigation

Isolating the Malformed Device Identifier

The isolation steps were:

1. Extract all identifiers from monitoring – query Prometheus for unique instance labels.

   $ curl -sG 'http://prometheus:9090/api/v1/label/instance/values' | jq -r '.data[]' | sort -u > /tmp/mon_instances.txt

2. Extract all identifiers from NetBox – use the NetBox REST API.

   $ curl -s -H "Authorization: Token $NETBOX_TOKEN" \
        http://netbox/api/dcim/devices/?limit=0 | \
        jq -r '.results[].name' | sort -u > /tmp/netbox_names.txt

3. Extract all identifiers from ServiceNow – via the Table API.

   $ curl -s -u "$SNOW_USER:$SNOW_PASS" \
        "https://instance.service-now.com/api/now/table/cmdb_ci_network?sysparm_limit=10000&sysparm_fields=name" | \
        jq -r '.result[].name' | sort -u > /tmp/snow_names.txt

4. Find the symmetric difference – names appearing in exactly one of the three sets.

   $ sort /tmp/mon_instances.txt /tmp/netbox_names.txt /tmp/snow_names.txt | uniq -u
   leaf-01-(prod)

The solitary entry confirmed the malformed identifier.

Code Examples for Identifier Validation

A reusable Python function that validates a device name against the platform convention:

import re

DEVICE_NAME_RE = re.compile(r'^[a-zA-Z0-9][a-zA-Z0-9\-_.]{0,62}$')

def is_valid_device_name(name: str) -> bool:
    """Return True if name conforms to the platform identifier policy."""
    return bool(DEVICE_NAME_RE.fullmatch(name))

# Example usage
if not is_valid_device_name("leaf-01-(prod)"):
    print("Invalid device name: leaf-01-(prod)")

The function can be imported into provisioning scripts, NetBox webhooks, and CI validation jobs.

CLI Commands for Device Identifier Correction

Once the bad name was identified, the correction was performed via the NetBox API (idempotent PATCH):

# Variables
NETBOX_URL="http://netbox/api"
TOKEN="$NETBOX_TOKEN"
OLD="leaf-01-(prod)"
NEW="leaf-01-prod"

# Find the device ID
DEVICE_ID=$(curl -s -H "Authorization: Token $TOKEN" \
    "$NETBOX_URL/dcim/devices/?name=$OLD&limit=1" | \
    jq -r '.results[0].id')

# Update the name
curl -s -X PATCH -H "Authorization: Token $TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"name\":\"$NEW\"}" \
    "$NETBOX_URL/dcim/devices/$DEVICE_ID/" | jq .

After the PATCH, the provisioning pipeline was re‑run with a sanitized site code, and the monitoring exporter was restarted to pick up the new sysName (changed via SNMP set):

snmpset -v2c -c private leaf-01-prod SNMPv2-MIB::sysName.0 s "leaf-01-prod"

Verification:

$ snmpget -v2c -c public leaf-01-prod SNMPv2-MIB::sysName.0
SNMPv2-MIB::sysName.0 = STRING: "leaf-01-prod"

Impact on Mitigation and Validation

Delayed Mitigation Efforts

Because the incident narrative was split:

• Mean Time To Acknowledge (MTTA) increased from the typical 3 min to 12 min (the time required for the NOC lead to notice the missing NetBox record).
• Mean Time To Resolve (MTTR) rose from 7 min to 28 min (the extra time spent correlating three separate evidence streams and performing the manual identifier correction).
• Post‑incident analysis showed duplicate work: two engineers independently opened a ticket in ServiceNow for the same symptom, and a third engineer opened a NetBox data‑quality ticket.

Changes to Follow‑up Validation Plan

The original validation plan relied solely on pre‑change connectivity tests (BGP, LLDP) executed via Nornir against the NetBox inventory. After the incident, the plan was expanded to include:

1. Identifier sanity check – a pre‑flight step that validates every device name against DEVICE_NAME_RE before inventory construction.
2. Cross‑system consistency verification – a job that compares the set of identifiers from NetBox, Prometheus, and ServiceNow and fails if the symmetric difference is non‑empty.
3. Automated remediation trigger – if a mismatch is detected, the job automatically creates a NetBox device record with a sanitized name (using a predefined mapping rule) and opens a change request for review.

Example Code for Revised Validation Script

#!/usr/bin/env python3
"""
validate_identifiers.py
- Pulls device names from NetBox, Prometheus, and ServiceNow.
- Ensures each name matches the platform regex.
- Ensures the three sets are identical (up to a known mapping table).
- Exits with non-zero status on any violation.
"""
import json, re, sys, os, requests

NETBOX_URL = os.getenv("NETBOX_URL")
NETBOX_TOKEN = os.getenv("NETBOX_TOKEN")
PROM_URL = os.getenv("PROMETHEUS_URL")
SNOW_INSTANCE = os.getenv("SNOW_INSTANCE")
SNOW_USER = os.getenv("SNOW_USER")
SNOW_PASS = os.getenv("SNOW_PASS")

DEVICE_NAME_RE = re.compile(r'^[a-zA-Z0-9][a-zA-Z0-9\-_.]{0,62}$')
SITE_MAP = {"(prod)": "-prod", "(dev)": "-dev"}  # simple sanitisation map

def fetch_netbox():
    headers = {"Authorization": f"Token {NETBOX_TOKEN}"}
    r = requests.get(f"{NETBOX_URL}/dcim/devices/?limit=0", headers=headers)
    r.raise_for_status()
    return {d["name"] for d in r.json()["results"]}

def fetch_prometheus():
    r = requests.get(f"{PROM_URL}/api/v1/label/instance/values")
    r.raise_for_status()
    return set(r.json()["data"])

def fetch_servicenow():
    auth = (SNOW_USER, SNOW_PASS)
    url = f"{SNOW_INSTANCE}/api/now/table/cmdb_ci_network?sysparm_limit=10000&sysparm_fields=name"
    r = requests.get(url, auth=auth)
    r.raise_for_status()
    return {rec["name"] for rec in r.json()["result"]}

def sanitize(name: str) -> str:
    for bad, good in SITE_MAP.items():
        name = name.replace(bad, good)
    return name

def main():
    nb = fetch_netbox()
    prom = fetch_prometheus()
    snow = fetch_servicenow()

    # Apply sanitisation to ServiceNow names for comparison
    snow_norm = {sanitize(n) for n in snow}

    all_names = nb | prom | snow_norm
    invalid = {n for n in all_names if not DEVICE_NAME_RE.fullmatch(n)}
    if invalid:
        print(f"Invalid device names: {invalid}", file=sys.stderr)
        sys.exit(1)

    # Check consistency
    if nb != prom or nb != snow_norm:
        print("Identifier sets diverge:", file=sys.stderr)
        print(f"  NetBox   : {nb}", file=sys.stderr)
        print(f"  Prometheus: {prom}", file=sys.stderr)
        print(f"  ServiceNow (norm): {snow_norm}", file=sys.stderr)
        sys.exit(2)

    print("All identifier checks passed.")
    sys.exit(0)

if __name__ == "__main__":
    main()

The script is invoked as a gate in the CI pipeline (pre-commit or pre-deploy) and as a nightly cron job for drift detection.

Scaling Limitations and Considerations

Identifier Uniqueness and Scalability

• The regex limits device names to 63 characters, sufficient for hierarchical naming (<site>-<role>-<num>-<env>) but restrictive when encoding additional metadata (e.g., VRF, tenant ID).
• In a multi‑tenant cloud‑scale deployment with > 10⁵ devices, the probability of accidental collisions rises if tenants are allowed to choose free‑form names. A tenant‑scoped namespace (e.g., <tenant-id>-<device-name>) is required to guarantee global uniqueness.

Tenant Isolation and Incident Containment

• When a malformed identifier appears in one tenant’s namespace, the cross‑system consistency check will still flag it because the sets are compared globally.
• To contain the blast radius, the validation job can be partitioned by tenant: each tenant runs its own identifier‑validation pipeline, and a