Which timestamp actually measures failover latency

Introduction to Convergence Benchmarking

Defining Convergence Metrics

Convergence is the interval from when a routing protocol detects a topology change to when the data plane forwards traffic over the new loop‑free path. We break it into three observable timestamps:

The convergence latency of interest is Δ = Tᵣ − Tₚ. In practice we observe Δ̂ = Tᵣ − Tₜ (telemetry‑to‑traffic) or Δ̃ = Tₜ − Tₚ (protocol‑to‑telemetry). Trustworthiness of each combination depends on clock synchronization, sampling jitter, and loss characteristics.

Importance of Reliable Timestamps

If any timestamp suffers from unbounded drift, variable latency, or unknown bias, the derived convergence metric is meaningless for regression testing. Reliable timestamps require:

1. Synchronized clocks across line cards, route processors, and telemetry collectors (PTP or NTP with bounded offset).
2. Deterministic observation points (e.g., a gNMI subscribe that timestamps at the agent, not at the collector).
3. Known sampling intervals and the ability to correlate a single event across streams (e.g., using a monotonic transaction ID).

Without these guarantees, a regression budget built on noisy convergence numbers will either over‑provision (wasting capacity) or under‑provision (risking black‑holes). The benchmark below makes these assumptions explicit and quantifies their impact.

Designing the Convergence Benchmark

Protocol Event Time Measurement

Instrument the protocol state machine so each transition emits a timestamped event. In modern NOS this is often available via model‑driven telemetry:

• BGP – /bgp/peer-state leaf changes from active → established.
• OSPF – /ospf/interface/state transitions (down → waiting → point‑to‑point).
• IS‑IS – /isis/adjacency/state (down → up).

If the vendor does not expose a native timestamp, augment the event with a local clock reading via a script that subscribes to the state change and records clock_gettime(CLOCK_REALTIME).

What the dashboard shows: A rising edge in the protocol state metric (e.g., bgp_peer_state{state="established"} == 1) at time Tₚ.
Limitations: The timestamp may not reflect the exact moment the protocol finished internal computation; there can be an internal processing delay between the state change and telemetry emission. Additional signals (internal protocol timers or per‑packet ASIC timestamps) are needed to bound this delay.

Telemetry Arrival Time Considerations

Telemetry arrival time is the timestamp attached when the observability platform ingests the sample. Two common models:

For a repeatable benchmark we must use the agent‑side timestamp and preserve it in a time‑series database (e.g., Prometheus with the __sample_timestamp__ metric).

What the dashboard shows: A sample appears in the telemetry stream at time Tₜ with a known monotonic offset from the agent’s clock.
Limitations: Does not guarantee the sample was not delayed inside the agent’s telemetry pipeline (e.g., queuing behind other subscriptions). Internal telemetry queue depth metrics are needed to bound this delay.

Traffic Restoration Time Tracking

Restoration is detected by measuring the first forward‑traffic packet that uses the new path. Approaches:

• Synthetic probe – Periodic UDP/TCP probe from a traffic generator; loss dropping to zero (or RTT returning to baseline) marks Tᵣ.
• In‑band telemetry – Flow‑based byte/packet counters on the egress interface of the new path; a non‑zero delta after a period of zero indicates restoration.
• Passive monitoring – Observe TCP retransmit timers or ACK gaps in flow records.

What the dashboard shows: A counter increase (e.g., ifOutOctets{ifname="eth0"}) that deviates from zero at time Tᵣ.
Limitations: Does not prove the traffic follows the intended new path; micro‑loops or ECMP hash changes could still deliver packets via an alternate route. Path verification (e.g., segment‑routing SID telemetry or OAM probe with TTL‑expired messages) is required for certainty.

Implementing the Benchmark

Protocol Event Time Collection (Python)

The following script subscribes to BGP peer state via gNMI, extracts the agent‑side timestamp from the response header, and exposes it as a Prometheus gauge.

#!/usr/bin/env python3
import time
import datetime
from gnmi.client import gNMIClient
from prometheus_client import start_http_server, Gauge

# Prometheus gauge: agent‑side timestamp of BGP peer state change (seconds since epoch)
bgp_event_ts = Gauge(
    'bgp_peer_event_timestamp_seconds',
    'Agent-side timestamp of BGP peer state change',
    ['peer_address', 'peer_as']
)

def bgp_state_handler(resp):
    for upd in resp.update:
        if upd.path.elem[-1].name == 'state':
            new_state = upd.val.json_val  # e.g., "ESTABLISHED"
            peer_addr = upd.path.key.get('peer-address')
            peer_as   = upd.path.key.get('peer-as')
            if new_state == 'ESTABLISHED':
                ts = resp.header.timestamp / 1e9  # ns → s
                bgp_event_ts.labels(peer_addr, peer_as).set(ts)
                print(
                    f"[{datetime.datetime.utcfromtimestamp(ts).isoformat()}] "
                    f"BGP peer {peer_addr}/{peer_as} ESTABLISHED"
                )

def main():
    start_http_server(8000)  # Exposes /metrics for Prometheus
    target = ('router1.example.com', 9339)
    with gNMIClient(
        target=target,
        username='admin',
        password='secret',
        insecure=True
    ) as client:
        subscribe = [
            {
                'path': [
                    'bgp',
                    'peer-state',
                    '*',   # wildcard for all peers
                    'state'
                ],
                'mode': 'sample',
                'sample_interval': 0  # send only on change
            }
        ]
        client.subscribe(subscribe=subscribe,
                         handler=bgp_state_handler,
                         timeout=None)

if __name__ == '__main__':
    main()

Result: Prometheus time series bgp_peer_event_timestamp_seconds{peer_address="10.0.0.1",peer_as="65001"} updates exactly when the agent observes the BGP state change.
Note: Internal BGP FSM processing time (between UPDATE receipt and state change) is not captured; it would require per‑packet ASIC timestamps or daemon tracepoints.

Telemetry Arrival Time Measurement (CLI)

Use gnmi_cli to pull a sample and capture the agent timestamp. The command below subscribes to OSPF interface state and prints the embedded timestamp.

# Install gnmi_cli: go install github.com/openhysys/gnmi/gnmi_cli@latest
gnmi_cli \
  -address router2.example.com:9339 \
  -username admin \
  -password secret \
  -insecure \
  -subs \
  -path "/ospf/interface-state[interface-name=eth0]/state" \
  -mode poll \
  -timeout 5s \
  -print_json \
  | jq '.update[0].val.json_val, .header.timestamp'

header.timestamp is nanoseconds since epoch set by the agent. To store it in Prometheus, run a tiny exporter:

#!/usr/bin/env bash
# ospf_telemetry_ts_exporter.sh
while true; do
  TS=$(gnmi_cli -address router2.example.com:9339 \
                -username admin -password secret -insecure \
                -subs -path "/ospf/interface-state[interface-name=eth0]/state" \
                -mode poll -timeout 2s -print_json \
                | jq -r '.header.timestamp')
  SEC=$(echo "scale=6; $TS/1e9" | bc)
  echo "ospf_telemetry_arrival_timestamp_seconds{ifname=\"eth0\"} $SEC"
  sleep 1
done

Run this script as a systemd service and point Prometheus at its /metrics endpoint.
Result: Gauge ospf_telemetry_arrival_timestamp_seconds jumps when the OSPF state change is sampled.
Note: Does not reveal internal telemetry queuing delay; a queue‑depth metric (e.g., gnmi_subscription_queue_depth) would be needed to bound it.

Traffic Restoration Time Monitoring (Script)

A lightweight traffic generator (iperf3 in UDP mode) measures loss; the script records the first second with zero loss.

#!/usr/bin/env bash
# traffic_restoration_monitor.sh
INTERFACE=eth0
REMOTE=10.0.0.2
PORT=5201
DURATION=30s   # total test window
INTERVAL=1s    # measurement interval

# Start iperf3 server (receiver) in background
iperf3 -s -1 > /dev/null &
SERVER_PID=$!

sleep 0.5   # allow server to bind
START=$(date +%s%N)   # ns epoch for reference
RESTORED=0

while [[ $(( ($(date +%s%N) - START) / 1000000000 )) -lt $DURATION ]]; do
  LOSS=$(iperf3 -c $REMOTE -u -b 100M -t $INTERVAL -J |
         jq '.end.sum.lost_percent')
  TS=$(date +%s%N)
  if (( $(echo "$LOSS == 0" | bc -l) )); then
    if [[ $RESTORED -eq 0 ]]; then
      RESTORED=1
      RESTORE_TS=$TS
      echo "traffic_restoration_timestamp_seconds $((RESTORE_TS/1000000000))"
    fi
  fi
  sleep $INTERVAL
done

kill $SERVER_PID 2>/dev/null

To expose the timestamp to Prometheus, a simple exporter can read the file written by the monitor:

#!/usr/bin/env bash
# restoration_exporter.sh
while true; do
  if [[ -f /var/run/restoration_ts.txt ]]; then
    TS=$(cat /var/run/restoration_ts.txt)
    echo "traffic_restoration_timestamp_seconds $TS"
  fi
  sleep 1
done

Result: Gauge traffic_restoration_timestamp_seconds jumps when loss drops to zero.
Note: Zero loss does not guarantee the traffic uses the intended new path; path verification (e.g., segment‑routing SID counters or OAM probes) is required for confirmation.

Data Analysis and Comparison

Correlating Timestamps for Trustworthiness

Once the three series are in Prometheus, align them using the monotonic agent clock. The following PromQL queries compute observable latency components and their variance.

Telemetry‑to‑traffic (Tᵣ − Tₜ):

traffic_restoration_timestamp_seconds
-
max_over_time(ospf_telemetry_arrival_timestamp_seconds[5m])

Protocol‑to‑telemetry (Tₜ − Tₚ):

ospf_telemetry_arrival_timestamp_seconds
-
bgp_peer_event_timestamp_seconds{peer_as="65001"}

Variance of Tᵣ − Tₜ over the last 10 minutes (indicates stability):

variance_over_time(
  traffic_restoration_timestamp_seconds
  -
  max_over_time(ospf_telemetry_arrival_timestamp_seconds[5m])
  [10m:]
)

If the variance stays below a chosen threshold (e.g., < 1 ms²), the corresponding timestamp combination can be considered trustworthy for routing regression budgets. High variance flags excessive jitter, unsynchronized clocks, or unpredictable processing delays, prompting further instrumentation (internal protocol timers, telemetry queue depth, or path‑verification signals).