Minimal Telemetry Sets for Correlated Incident Replay

Introduction to Network Incident Replay

Accurate replay lets engineers reconstruct the exact sequence of control‑plane, data‑plane, and telemetry events that led to a service‑impacting condition. Without faithful replay, root‑cause analysis relies on inference and can miss transient interactions such as micro‑bursts that trigger queue drops only observable at sub‑second granularity.

A data‑first entry point is a timeline built from three correlated streams: interface counters, BGP state changes, and application‑level latency spikes. The following PromQL query returns a non‑zero vector when any of these signals changes in the last 5 minutes:

(
  increase(ifInOctets[5m]) 
  or 
  increase(bgpPeerStateChanges[5m]) 
  or 
  histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
) 
> 0

If the vector is empty, the incident left no trace in these metrics, indicating a blind spot (e.g., micro‑burst loss not captured by 5‑minute counters).

Benefits of Combining Logs, Span Tags, and Event Topics

Together they provide immutable, timestamped records (logs), causal identifiers (span tags), and high‑volume, low‑latency state‑change notifications (event topics) that enable faithful reconstruction of network incidents.

Understanding Log Collection

Types of Logs for Incident Replay

Log Format and Structure Considerations

• Structured JSON (preferred) enables direct ingestion into Loki, Elasticsearch, or Splunk without regex parsing. Example:

  {
    "timestamp": "2025-10-31T14:22:07.123Z",
    "host": "router01",
    "facility": "local0",
    "severity": "info",
    "msg": "BGP peer 10.0.0.2 changed state from Established to Idle",
    "bgp": {
      "peer": "10.0.0.2",
      "old_state": "Established",
      "new_state": "Idle",
      "reason": "Hold timer expired"
    }
  }

  The dedicated bgp object allows a query like | json | bgp.peer == "10.0.0.2" to isolate all BGP‑related log lines without full‑text scanning.

• Unstructured plain text requires regex pipelines; risk of missing fields if the format drifts.

• Timestamp precision – millisecond or microsecond resolution is essential for sub‑second correlation with spans. Second‑granular logs cannot determine whether a log preceded or followed a span that started the same second.

Log Collection Tools and Technologies

Missing signal: Collecting only syslog without structured fields loses the ability to filter by trace_id or bgp.peer without costly regexes, increasing backend cardinality risk.

Span Tags and Distributed Tracing

Introduction to Span Tags and Distributed Tracing

A span represents a single unit of work (e.g., an RPC, a database query, a packet processing step). Span tags are key‑value pairs attached to a span that enrich it with contextual data (e.g., error=true, interface=eth0, bgp.prefix=10.0.0.0/24).

Data‑first entry point – a trace view showing latency breakdown for a failed BGP update:

Trace ID: a1b2c3d4e5f6
Span ID: 0x9f8e7d6c
Operation: bgp_update_send
Start: 2025-10-31T14:22:07.123456Z
Duration: 2.3ms
Tags:
  - peer=10.0.0.2
  - prefix=10.0.0.0/24
  - error=true
  - error.code=hold_timer_expired

The error=true tag appears on the span that actually transmitted the update, pinpointing where the hold‑timer expiration was detected. Without this tag, you would only see a generic latency spike.

Using Span Tags for Incident Replay

1. Inject trace context into network‑device telemetry (e.g., gNMI subscriptions) using the OpenTelemetry SDK.
2. Correlate each span’s trace_id with log lines containing the same identifier (added by the logging library).
3. Replay by re‑executing the span tree in a trace visualizer (Jaeger, Tempo) while feeding the original log stream as overlay events.

Example: A gNMI subscription that streams interface counters can be annotated with a span:

// Pseudo‑Go using OpenTelemetry
ctx, span := tracer.Start(ctx, "gNMI_subscribe_interface")
defer span.End()
span.SetAttributes(
    attribute.String("interface", "eth0"),
    attribute.Int64("subscription_id", 42),
)
// inside the subscription loop:
for {
    resp, err := client.Subscribe(ctx, req)
    // record each response as an event
    span.AddEvent("gnmi_response",
        attribute.Int64("timestamp", time.Now().UnixNano()),
        attribute.String("json_val", string(resp.GetUpdate())))
    // ...
}

Each response becomes an event attached to the span, allowing you to replay the exact sequence of counter updates alongside logs that reference the same subscription_id.

Popular Distributed Tracing Tools and Frameworks

Missing signal: Capturing spans only at the application layer omits device‑level latency (e.g., ASIC pipeline stalls) visible in gNMI streams.

Event Topics and Messaging Systems

Introduction to Event Topics and Messaging Systems

An event topic is a named channel in a publish/subscribe system where producers emit immutable events and consumers receive them in order (or with configurable guarantees). For network incident replay, topics carry:

• Interface state changes (ifOperStatus)
• BGP peer events (bgpPeerStateChanged)
• Telemetry samples (ifInOctets, cpuUtilization)
• Configuration commits (netconfCommit)

Because the messaging system preserves the exact byte payload and ordering (when configured for strong ordering), replaying the topic yields a bit‑for‑bit reconstruction of the original event stream.

Using Event Topics for Incident Replay

1. Produce events directly from the source (e.g., a gNMI telemetry collector publishes each JSON update to a Kafka topic).
2. Consume the topic during replay, feeding the events into a deterministic replay engine (e.g., a custom Go program that re‑applies config changes, updates a mock routing table, and emits synthetic logs/spans).
3. Correlate each event’s timestamp with log entries and span timestamps to align the three streams.

Data‑first entry point – a Kafka consumer showing the exact sequence of events leading to a link flap:

$ kafka-console-consumer \
    --bootstrap-server kafka01:9092 \
    --topic network-events \
    --from-beginning \
    --property print.key=true \
    --property print.value=true \
    --max-messages 5

Output (reproduced lab output):

key:   ifOperStatus/eth0
value: {"timestamp":"2025-10-31T14:22:05.001Z","ifName":"eth0","oldState":"up","newState":"down","reason":"linkLoss"}
key:   bgpPeerStateChanged
value: {"timestamp":"2025-10-31T14:22:05.012Z","peer":"10.0.0.2","oldState":"Established","newState":"Idle","holdTimeExpired":true}
key:   telemetry/ifInOctets
value: {"timestamp":"2025-10-31T14:22:05.020Z","ifName":"eth0","value":0}
key:   log
value: {"timestamp":"2025-10-31T14:22:05.030Z","host":"router01","msg":"Link down on eth0, L1 loss detected"}
key:   span
value: {"traceID":"a1b2c3d4e5f6","spanID":"0x9f8e7d6c","name":"link_down_handling","timestamp":"2025-10-31T14:22:05.040Z","durationMs":1.2,"tags":{"interface":"eth0","error":true}}

The timestamps are strictly increasing (monotonic) and each event type appears in causal order: link loss → BGP state change → traffic drop → log entry → handling span. Missing or out‑of‑order events would cause the replay to diverge from the real incident.

Popular Messaging Systems and Event Topic Implementations

End of document.