Rollback gates when telemetry lags the change

Introduction to Rollback State Comparison

Intended Rollback State

The intended rollback state is the configuration that an automation system (Ansible, Terraform, NetOps platform, etc.) calculates as the desired configuration after a rollback. It is derived from the source‑of‑truth (SoT) repository, the original commit hash, or a stored snapshot representing the known‑good baseline.

# Timestamp of the last successful commit stored in the SoT
sot_last_commit_timestamp{device="router01"}

If this metric is missing, operators must infer intent from change‑request tickets or CI/CD logs, introducing latency and blind spots.

Device‑Confirmed State

Device‑confirmed state is the configuration that the network element reports as active after processing a configuration transaction. It can be obtained via model‑driven telemetry (gNMI Get), CLI show running-config, or NETCONF <get-config>.

# gNMI Get response encoded as a metric for a specific leaf
device_config_leaf_value{device="router01", path="/interfaces/interface[name=eth0]/mtu", value="1500"}

When the device does not export the full config via telemetry, operators rely on periodic polling, creating a gap between application and visibility.

Delayed Telemetry

Delayed telemetry refers to streaming operational data (interface counters, BGP peer state, config‑change notifications) that arrives at the collector out of order relative to the transaction timeline. Causes include network retransmission, exporter buffering, or varying serialization delays.

# Stream of config‑change notifications with event‑time and processing‑time timestamps
gnmi_subscribe_event_time{device="router01", path="/interfaces"}
gnmi_subscribe_process_time{device="router01", path="/interfaces"}

A divergence between event_time and process_time signals out‑of‑order delivery. Without both timestamps, detecting reordering requires external packet captures.

Challenges in Comparing States

Impact of Out‑of‑Order Streaming Signals

When telemetry arrives out of order, a simple comparison of the latest received state to the intended state can yield false positives or negatives.

# Detect a rollback where device‑confirmed state temporarily appears newer than intended
max_over_time(device_config_leaf_value[5m]) 
  > sot_intended_config_value{device="router01", path="/interfaces/interface[name=eth0]/mtu"}

If device_config_leaf_value is subject to reordering, the max_over_time window may spike, suggesting a drift that does not persist. Without per‑event sequencing information, operators cannot distinguish a genuine mis‑configuration from a telemetry artifact.

Consequences of Aborting Healthy Changes or Continuing Bad Ones

Aborting a healthy change based on a spurious drift signal causes unnecessary service impact, wasted rollback windows, and erodes trust in automation. Conversely, continuing a bad change because telemetry has not yet reflected the failure can prolong packet loss, blackholing, or security exposure.

# Alert on persistent mismatch for >2 minutes
absent_over_time(device_config_leaf_value[2m]) 
  or (device_config_leaf_value != sot_intended_config_value)

If the alert fires due to delayed telemetry, a good change may be aborted; if it fails to fire because stale data hides the mismatch, a bad change persists. The root issue is the lack of a transaction identifier that ties a specific config push to its corresponding telemetry events.

Technical Approach to State Comparison

Data Structures for State Representation

Each configuration element is represented as a tuple:

(key, value, intent_version, device_version, event_time, process_time)

If any field is unavailable (commonly event_time/process_time in legacy SNMP), annotate it as unknown and treat missing timestamps as a source of uncertainty in the comparison algorithm.

Algorithms for Comparing Intended and Confirmed States

1. Three‑way diff with version vectors

   • Load intended state map I[key] = (value, intent_version).
   • Load device‑confirmed state map D[key] = (value, device_version, event_time, process_time).
   • For each key:
     • If I[key].value == D[key].value and I[key].intent_version <= D[key].device_version → in sync.
     • If values differ → drift (requires further inspection).
     • If I[key].intent_version > D[key].device_version → pending (device has not yet applied).
   • Buffer pending items until a watermark (e.g., max observed process_time minus a safety lag) passes; treat unresolved pending items as drift.

2. Out‑of‑order tolerant merge (CRDT‑inspired)

   • Maintain a per‑key observed‑set of (value, version, event_time).
   • Apply a last‑write‑wins rule based on event_time when multiple versions arrive.
   • Compare the resulting convergent value to the intent value using intent_version as a tie‑breaker.

   If event_time is missing, fall back to process_time and increase the safety lag, which reduces detection latency but raises false‑positive risk.

Handling Delayed Telemetry in State Comparison

• Event‑time windowing: Buffer incoming notifications for a configurable window (e.g., 10 s) before emitting a comparison result.

• Watermarking: Compute watermark = max(event_time) - allowed_lateness. Only keys with all notifications older than the watermark are considered for final comparison.

• Re‑ordering detection metric:

  # Ratio of out‑of‑order notifications per device
  rate(gnmi_reorder_detected_total[5m]) / rate(gnmi_notifications_total[5m])

  A rising ratio indicates that window or lateness parameters need tuning.

• Fallback to polling: When streaming telemetry shows high reorder rates, temporarily increase the frequency of gNMI Get or NETCONF <get-config> to ground‑truth the state.

If the system lacks explicit event_time/process_time fields, rely on heuristic timeouts—document this as a limitation because it cannot guarantee correctness.

Troubleshooting State Comparison Issues

Identifying Symptoms of Out‑of‑Order Signals

• Flapping drift alerts that clear within seconds without operator action.
• Sudden spikes in gnmi_reorder_detected_total coinciding with configuration pushes.
• Discrepancy between show running-config (CLI) and the latest telemetry value for the same leaf.

# Alert on alert flapping (>3 state changes in 1 min)
changes(alertstate{alertname="ConfigDriftDetected"}[1m]) > 3

If the alerting system does not expose alertstate, manually inspect alert logs, increasing mean time to detect (MTTD).

Debugging Techniques for State Comparison

1. Enable per‑message tracing in the gNMI exporter (if supported) to embed a unique correlation ID that is also logged by the collector.

2. Capture raw gNMI packets on the exporter side:

   tcpdump -w gnmi.pcap port 9339

   Compare timestamps with collector logs.

3. Replay a known‑good transaction in a lab with deterministic network delay (e.g., tc netem) to verify algorithm convergence.

4. Compare version vectors: Export intent_version and device_version as labels and plot their divergence over time.

If the exporter does not support custom correlation IDs, fall back to source IP/port tuples—less reliable in NAT‑ed environments.

Common Pitfalls in Implementing State Comparison

• Wall‑clock synchronization assumptions: Using the collector clock as the sole basis for ordering can produce false drift if devices and collectors are not NTP‑synced to < 1 ms.
• Ignoring schema drift: Comparing raw values without validating that the YANG schema version matches between SoT and device can mask semantic differences (e.g., a leaf renamed in a newer model).
• Over‑reliance on last‑received value: Treating the most recent telemetry sample as ground truth ignores the possibility that an older sample is actually more correct due to reorder.
• Missing transaction boundaries: Without a commit ID, it is impossible to know which subset of a large config change caused a particular drift signal.

Each pitfall should be explicitly called out in design documents and mitigated via versioned schemas, NTP monitoring, and transaction‑ID propagation.

Code Examples for State Comparison

CLI Examples for Manual State Comparison

# 1. Retrieve intended state from Git (assuming a commit hash)
git show <commit-hash>:router01/intended.cfg > /tmp/intended.cfg

# 2. Pull current config via gNMI Get
gnmi_get -addr router01:9339 -timeout 10s \
    -path "/interfaces/interface[name=eth0]/mtu" > /tmp/current.cfg

# 3. Normalize both files (strip whitespace, sort)
cat /tmp/intended.cfg | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | sort > /tmp/intended.norm
cat /tmp/current.cfg | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | sort > /tmp/current.norm

# 4. Diff
diff -u /tmp/intended.norm /tmp/current.norm

If the device does not support gNMI Get, replace step 2 with a CLI scrape:

ssh router01 "show running-config | include mtu" > /tmp/current.cfg

Note that CLI output may require additional parsing due to formatting differences.

Programming Language Examples for Automated State Comparison

Python (using protobuf for gNMI and deepdiff for diff):

import grpc
import time
import threading
from gnmi import gnmi_pb2, gnmi_pb2_grpc
from deepdiff import DeepDiff
import json

def get_intent_state(commit_hash):
    """
    Load intended state from Git or a key‑value store.
    Returns a dict mapping YANG path strings to their values.
    """
    # Placeholder implementation
    return {}

def subscribe_gnmi(stop_event):
    """
    Yield gNMI Subscribe responses until stop_event is set.
    """
    channel = grpc.insecure_channel('router01:9339')
    stub = gnmi_pb2_grpc.gNMIStub(channel)
    subscribe = gnmi_pb2.SubscribeRequest(
        subscribe=gnmi_pb2.SubscriptionList(
            subscription=[
                gnmi_pb2.Subscription(
                    path="/interfaces/interface[name=eth0]/mtu",
                    mode=gnmi_pb2.SubscriptionMode.STREAM,
                )
            ],
            mode=gnmi_pb2.SubscriptionList.STREAM,
        )
    )
    for resp in stub.Subscribe(iter([subscribe])):
        if stop_event.is_set():
            break
        yield resp

def compare_states(intent, telemetry_stream):
    """
    Maintain device state from telemetry and compare against intent.
    """
    device_state = {}  # path -> {'value': ..., 'event_time': ..., 'process_time': ...}
    for update in telemetry_stream:
        for n in update.update:
            # Simplified path extraction; adapt to full YANG path handling
            path = n.path.elem[-1].name
            value = json.loads(n.val.json_val)
            device_state[path] = {
                "value": value,
                "event_time": update.timestamp,
                "process_time": int(time.time() * 1e9),
            }

        # Emit comparison after watermark logic (omitted for brevity)
        current_values = {k: v["value"] for k, v in device_state.items()}
        diff = DeepDiff(intent, current_values)
        if diff:
            print("Drift detected:", diff)
        else:
            print("In sync")

if __name__ == "__main__":
    intent = get_intent_state("a1b2c3d")
    stop = threading.Event()
    t = threading.Thread(target=compare_states, args=(intent, subscribe_gnmi(stop)))
    t.start()
    # Run for some time, then stop
    time.sleep(60)
    stop.set()
    t.join()

This script subscribes to a gNMI stream, builds a device‑state map, and periodically diffs it against the intended state using deepdiff. Extend it with watermarking, version‑vector checks, and re‑ordering detection as needed for production use.