Drift timelines from config and telemetry

Introduction

Correlating commit logs, gNMI snapshots, and reachability signals lets you pinpoint when configuration drift first appeared and verify whether a reconciliation action fixed the intended issue. Commit logs provide the ground‑truth of intent, gNMI snapshots reveal the actual device state, and reachability probes confirm whether the data‑plane behaves as expected.

Overview of Commit Logs

Commit logs are immutable records of intentional changes, typically sourced from a version‑control system (Git, SVN) or an orchestrator that writes a candidate transaction and pushes it to the running datastore. Each entry includes:

Timestamp – when the commit was applied (usually UTC).
Actor – user or automation service that initiated the change.
Message – free‑form description or ticket reference.
Diff – the set of YANG‑modeled nodes added, modified, or deleted (unified diff or JSON patch).

From an observability standpoint the commit log is the source of truth for intent. It does not guarantee that the device applied the change (validation failures, rollbacks, hardware faults) nor does it capture transient states between the commit and the next polling interval.

Understanding gNMI Snapshots

gNMI (gRPC Network Management Interface) provides a standardized Get operation that returns a point‑in‑time snapshot of the YANG datastore as a JSON‑encoded protobuf message. Typical snapshot contents:

Interface operational state (if:oper-status, if:phys-address).
BGP peer state (bgp:peer-state).
QoS policy counters.
Config‑hash leaf (if modeled by the vendor).

A snapshot is self‑describing: each leaf carries its YANG path, enabling exact diff‑based comparison with another snapshot or with the intended state derived from a commit.
It reflects only the data the device exposes at request time; control‑plane crashes, stale sessions, or sampling gaps can produce incomplete or outdated views.

Role of Reachability Signals in Network Monitoring

Reachability signals are active‑probe metrics answering “Can I reach X from Y?”. Common implementations:

ICMP echo (ping) latency and loss.
TCP SYN‑handshake success rate to a service port.
BFD session state (up/down, detect‑time).
Synthetic HTTP GET success/failure.

These probes are independent of device telemetry; they monitor the forwarding path from an external observer (e.g., a monitoring host or dedicated probe node). When combined with commit logs and gNMI snapshots they reveal whether a configuration change actually altered user‑visible behavior.
Reachability does not disclose the root cause of a failure (ACL drop vs. hardware fault) nor provide internal state such as queue depths unless explicitly probed.

Correlating Commit Logs and gNMI Snapshots

Collecting and Parsing Commit Logs

Commit logs are ingested via a Git webhook or periodic git pull. A lightweight parser extracts the SHA, timestamp, author, and unified diff, converting the diff into a set of intended YANG path‑value pairs.

Example parsed commit log entry (JSON)

{
  "commit": "a3f9c2e",
  "timestamp": "2025-09-24T14:02:11Z",
  "author": "netops@example.com",
  "message": "Add BGP peer 10.1.2.3 AS 65012",
  "intended_state": [
    { "path": "/bgp/neighbor[peer-address='10.1.2.3']", "op": "set", "value": { "peer-as": 65012 } },
    { "path": "/bgp/neighbor[peer-address='10.1.2.3']/timers", "op": "set", "value": { "hold-time": 180 } }
  ]
}

The intended_state array lists the exact YANG paths and values the commit claims to enforce—this becomes the baseline for snapshot comparison.

Capturing and Analyzing gNMI Snapshots

A snapshot is obtained via a gNMI Get request. The response contains Notification messages with Update elements; flattening yields a map of path → value.

Example gNMI GetResponse snippet

{
  "notification": [
    {
      "timestamp": 1737775331000000000,
      "update": [
        {
          "path": { "elem": [
            { "name": "bgp" },
            { "name": "neighbor" },
            { "key": { "peer-address": "10.1.2.3" } },
            { "name": "peer-as" }
          ] },
          "val": { "uint_val": 65012 }
        },
        {
          "path": { "elem": [
            { "name": "bgp" },
            { "name": "neighbor" },
            { "key": { "peer-address": "10.1.2.3" } },
            { "name": "timers" },
            { "name": "hold-time" }
          ] },
          "val": { "uint_val": 180 }
        }
      ]
    }
  ]
}

Matching values indicate the device applied the change correctly at snapshot time. A mismatch (e.g., missing or zero peer-as) flags a configuration‑drift candidate.

Techniques for Correlating Logs and Snapshots

Timestamp alignment – Convert commit timestamps to epoch nanoseconds and compare with the gNMI notification timestamp. A drift is suspect if the snapshot is after the commit but the values do not reflect the commit.
Path‑based diff – Compute set differences between intended_state (from commit) and observed_state (from snapshot).
- added = paths in intended not observed → likely not applied.
- removed = paths observed not intended → leftover state or rollback.
Version vector – If the vendor exposes a config-revision leaf, compare the commit‑revision (often embedded in the commit message) with the snapshot’s revision; a mismatch is a strong drift indicator.

Pseudo‑code for correlation (Python‑like)

def correlate(commit, snapshot):
    intended = { (c['path'], json.dumps(c['value'])) for c in commit['intended_state'] }
    observed = { (u['path'], json.dumps(u['value'])) for u in snapshot['updates'] }
    missing = intended - observed
    extra   = observed - intended
    return missing, extra

Non‑empty missing indicates the commit did not take effect; non‑empty extra hints at stray configuration.

Integrating Reachability Signals for Enhanced Insights

Understanding Reachability Signal Metrics

Reachability is exported as time‑series metrics (Prometheus convention examples):

probe_success{target="10.1.2.3", module="icmp"} – 1 if the probe succeeded, 0 otherwise.
probe_latency_seconds{target="10.1.2.3", module="tcp"} – RTT of a TCP SYN.
bgp_peer_state{peer="10.1.2.3", state="established"} – 1 if BGP is up.

These metrics are generated by an external probe (e.g., blackbox_exporter or a custom BFD agent) and are independent of the device’s internal gNMI telemetry.
If no probe mirrors the traffic class affected by a configuration change (e.g., MPLS LSP ping for a label‑swap), you cannot directly verify data‑plane impact.

Combining Reachability Data with Commit Logs and gNMI Snapshots

Create a unified timeline where each event (commit, snapshot, probe result) shares the same time axis:

Commit logs → ingest as annotations (e.g., Prometheus alertname="config_commit" with value=1 at commit time).
gNMI snapshots → store as gauges per path (e.g., bgp_neighbor_peer_as{peer="10.1.2.3"}).
Probe metrics → export normally.
Dashboard → overlay:
- Vertical lines at commit timestamps.
- Line graphs of relevant gNMI gauges.
- Bar chart of probe success.

Example PromQL: BGP peer AS vs. probe success

# BGP peer AS from gNMI (gauge)
bgp_neighbor_peer_as{peer="10.1.2.3"}
#
# Probe success (blackbox)
probe_success{target="10.1.2.3", module="tcp"}

When bgp_neighbor_peer_as steps from 0 to 65012, probe_success should transition from 0 to 1 within a few seconds if the change is effective. A delay or persistent 0 indicates a problem.

Example Code for Data Integration

A Bash script that pulls the latest commit, triggers a gNMI Get, queries Prometheus for probe success, and prints a correlation table.

#!/usr/bin/env set -euo pipefail
REPO="/opt/netconfig/repo"
DEVICE="router01.example.com"
PEER="10.1.2.3"

# 1. Latest commit
cd "$REPO"
COMMIT=$(git rev-parse --short HEAD)
COMMIT_TIME=$(git show -s --format=%ct HEAD)   # epoch seconds

# 2. gNMI Get (using gnmic)
SNAP=$(gnmic get --address "$DEVICE":9339 --username admin --password secret \
    --path "/bgp/neighbor[peer-address='$PEER']/peer-as" --encoding json)

AS_VAL=$(echo "$SNAP" | jq -r '.notification[0].update[0].val.uint_val // empty')

# 3. Probe success from Prometheus
PROBE=$(curl -sG "http://prometheus:9090/api/v1/query" \
    --data-urlencode 'query=probe_success{target="'$PEER'",module="tcp"}' |
    jq -r '.data.result[0].value[1] // "0"')

printf "Commit %s @ %s\n" "$COMMIT" "$(date -d @"$COMMIT_TIME" --iso-8601=seconds)"
printf "gNMI peer-as: %s\n" "${AS_VAL:-<missing>}"
printf "Probe success: %s\n" "$PROBE"

In production, push each datum to a time‑series database and rely on its query language for alignment rather than manual scripting.

Identifying Drift through Correlated Data

Detecting Configuration Drift

Configuration drift appears as a persistent mismatch between intended_state (derived from commits) and observed_state (from gNMI) that survives multiple snapshot intervals.

Detection rule (PromQL)

# Inject intended AS as a custom metric from the commit processor
bgp_intended_peer_as{peer="10.1.2.3"}
#
# Observed AS from gNMI
bgp_observed_peer_as{peer="10.1.2.3"}
#
# Drift flag: 1 when they differ
drift = (bgp_intended_peer_as != bgp_observed_peer_as)

The drift time series is 0 during conformity and flips to 1 when a mismatch exists; its duration reveals how long the drift persisted.

Analyzing Network State Drift

Network‑state drift encompasses transient conditions (flapping interfaces, QoS mis‑application, routing loops) that may not appear in a static config diff. Detect it by examining behavioral metrics that should remain stable given the intended configuration.

Example: After committing a QoS policy that sets dscp=46 on a class, expect the metric if_out_dscp_46_packets to rise proportionally to traffic. If the packet counter stays flat while if_out_octets increases, the policy is not being applied—indicating network‑state drift despite a clean config diff.

By continuously correlating commit intent, gNMI observed state, and reachability probes, you can pinpoint the exact moment drift emerges, verify whether a reconciliation action corrected the underlying issue, and distinguish between pure configuration errors and broader data‑plane anomalies.