Finding a one-segment outage in green dashboards

Introduction

In a multi‑tenant data‑center fabric, global tunnel counts and spine‑level health (CPU, memory, link utilization < 30 %) can appear normal while tenants experience intermittent packet loss and latency spikes. This article shows how to detect partial tenant degradation—when only a subset of tunnels for a specific tenant is impaired—by using tenant‑scoped telemetry, correlated timelines, and the signals that separate local damage from fabric‑wide failure.

Understanding Tenant‑Specific Telemetry

Tenant‑specific telemetry is any measurement tagged or scoped to a logical tenant construct (VRF, VPN‑ID, tenant‑ID, etc.). In model‑driven telemetry this is achieved by:

• Adding tenant‑identifying keys to the data model (e.g., augmenting ietf-te:tunnel with tenant-id).
• Exporting per‑tenant counters via gNMI subscriptions on paths such as /network-instances/network-instance[name=Tenant-A]/protocols/protocol[bgp]/bgp/peer.
• Streaming per‑tenant queue depth, latency histograms, and loss ratios from line‑cards or switches.

If the platform does not natively export tenant‑scoped counters, they can be derived by post‑processing raw interface or tunnel metrics with the tenant‑to‑tunnel mapping (often available from the orchestrator or IPAM).

Missing signal note: When only aggregate tunnel counters are exported without a tenant label, per‑tenant loss must be inferred from flow‑sampling or sFlow data, which introduces sampling error.

Importance of Correlated Timelines

A correlated timeline aligns multiple data streams on a common time axis, revealing cause‑effect relationships invisible when each stream is viewed in isolation. For tenant degradation detection the timeline typically includes:

1. Tenant‑scoped metrics (latency, loss, jitter, tunnel flap count).
2. Global fabric health (spine CPU, link utilization, BGP peer state).
3. Application logs/request traces (timestamped latency or error codes).
4. Control‑plane events (configuration changes, ACL updates, QoS policy pushes).

Overlaying these streams lets you answer questions such as:

• Did the latency spike coincide with a change in the tenant’s QoS policy?
• Did a burst of micro‑bursts on a specific leaf‑to‑leaf link appear only for tunnels belonging to the affected tenant?

If fabric‑wide events show no correlation with tenant‑specific symptoms, the problem is likely local to the tenant’s data path.

Identifying Partial Tenant Degradation

Analyzing Global Tunnel Counts and Spine Health

Global tunnel count (tunnel_up_total) and spine health metrics (e.g., spine_cpu_utilization, spine_link_utilization) are aggregate signals. They mask heterogeneity because:

• A small subset of troubled tunnels can be diluted by thousands of healthy tunnels.
• Spine CPU may stay low even if a few leaf‑to‑leaf paths are congested, as the spine forwards traffic in hardware and sees only packet counts, not per‑flow queuing.

PromQL example (assuming the metrics exist):

# Global tunnel up count – looks stable
sum by (job) (tunnel_up_total)

# Spine CPU – also stable
avg by (instance) (spine_cpu_utilization)

Both queries return flat lines during an incident, giving a false sense of health.

Limitations of Global Metrics

Role of Tenant‑Scoped Telemetry

Tenant‑scoped telemetry provides the granularity needed to see the tail of the distribution. Key metrics include:

• tenant_tunnel_latency_seconds{tenant="Tenant-A", direction="egress"} – histogram or summary.
• tenant_tunnel_loss_ratio{tenant="Tenant-A"} – packets lost / packets sent.
• tenant_queue_depth_packets{tenant="Tenant-A", queue="priority-0"} – depth per priority queue.
• tenant_tunnel_flap_count{tenant="Tenant-A"} – number of state changes per interval.

When any of these diverge from baseline while global metrics stay nominal, you have a partial tenant degradation signal.

Tenant‑Scoped Telemetry and Metrics

Overview of Relevant Metrics

If the platform does not export these metrics directly, compute them from raw counters using the tenant‑to‑tunnel mapping (see CLI examples below).

Configuring and Collecting Tenant‑Scoped Telemetry

Model‑driven approach (gNMI)

1. Define a subscription that includes tenant‑identifying keys. Example gNMI path (JSON‑encoded):

{
  "subscription": [
    {
      "path": {
        "elem": [
          { "name": "network-instances" },
          { "name": "network-instance", "key": { "name": "Tenant-A" } },
          { "name": "protocols" },
          { "name": "protocol", "key": { "identifier": "bgp", "name": "default" } },
          { "name": "bgp" },
          { "name": "peer" },
          { "name": "state" }
        ]
      },
      "mode": "SAMPLE",
      "sample_interval": 10000000000   // 10 s in nanoseconds
    },
    {
      "path": {
        "elem": [
          { "name": "interfaces" },
          { "name": "interface", "key": { "name": "tunnel100" } },
          { "name": "counters" }
        ]
      },
      "mode": "SAMPLE",
      "sample_interval": 5000000000   // 5 s
    }
  ],
  "encoding": "PROTO",
  "heartbeat_interval": 30000000000   // 30 s
}

2. Start the subscription using gnmic:

gnmic -a leaf1:9339 -u admin -p \
    --encoding proto \
    subscribe \
    --subscription-path ./tenant-a-subscription.json

The output can be piped to a Prometheus remote‑write endpoint or a time‑series database (e.g., Cortex, Thanos) for downstream querying.

CLI‑based collection (when gNMI not available)

On many NOS platforms you can fetch per‑tenant tunnel stats via CLI and push them to a monitoring system via a script. Example on a hypothetical netos CLI:

# 1. Get list of tunnels belonging to Tenant-A
show tunnel brief | include Tenant-A | awk '{print $1}' > /tmp/tenant-a-tunnels.txt

# 2. For each tunnel, collect latency and loss
while read tun; do
  latency=$(show tunnel $tenu detail | grep "Avg Latency" | awk '{print $3}')
  loss=$(show tunnel $tenu detail | grep "Packet Loss" | awk '{print $3}')
  echo "tenant_tunnel_latency_seconds{tenant=\"Tenant-A\",tunnel=\"$tun\"} $latency"
  echo "tenant_tunnel_loss_ratio{tenant=\"Tenant-A\",tunnel=\"$tun\"} $loss"
done < /tmp/tenant-a-tunnels.txt

Run the script every 30 s via cron and push the formatted lines to a Pushgateway or directly to Prometheus via the textfile collector.

Example CLI Commands for Data Collection

Correlating Timelines for Root Cause Analysis

Understanding Correlated Timelines

A correlated timeline is a time‑aligned multi‑series plot where each series represents a different telemetry dimension. Its power lies in visual inspection (or algorithmic correlation) to detect:

• Temporal precedence – does a change in metric A lead to a change in metric B?
• Co‑occurrence – do two anomalies appear within the same window?
• Absence of correlation – when fabric metrics stay flat while tenant metrics deviate, indicating a local issue.

Tools and Techniques

Example Code for Automating Timeline Correlation

import requests
import pandas as pd
import numpy as np

PROM_URL = "http://prometheus:9090/api/v1/query_range"

def fetch_series(query, start, end, step="5s"):
    params = {"query": query, "start": start, "end": end, "step": step}
    r = requests.get(PROM_URL, params=params)
    r.raise_for_status()
    data = r.json()["data"]["result"]
    values = np.array([float(v[1]) for v in data[0]["values"]])
    times = pd.to_datetime([v[0] for v in data[0]["values"]], unit="s")
    return pd.Series(values, index=times)

# Time window of interest (e.g., last 2 h)
now = pd.Timestamp.utcnow()
start = now - pd.Timedelta(hours=2)
end = now

latency = fetch_series(
    'histogram_quantile(0.95, sum by (le,tenant) (rate(tenant_tunnel_latency_seconds_bucket{tenant="Tenant-A"}[30s])))',
    start.timestamp(), end.timestamp()
)

spine_cpu = fetch_series(
    'avg by (instance) (spine_cpu_utilization{job="spine"})',
    start.timestamp(), end.timestamp()
)

# Align on common index (inner join)
df = pd.DataFrame({"latency": latency, "spine_cpu": spine_cpu}).dropna()

# Cross‑correlation for lags -30..+30 steps (±2.5 min)
lags = np.arange(-30, 31)
corrs = [df["latency"].corr(df["spine_cpu"].shift(lag)) for lag in lags]
best_lag = lags[np.argmax(np.abs(corrs))]
best_corr = np.max(np.abs(corrs))

print(f"Best lag: {best_lag} steps ({best_lag*5}s) with correlation {best_corr:.3f}")
if best_lag < 0:
    print("Latency leads spine CPU (possible local issue).")
else:
    print("Spine CPU leads latency (fabric‑wide effect).")

Interpretation: A negative lag (tenant latency changes before spine CPU) with moderate correlation suggests the latency spike is not driven by spine load but by something local to the tenant’s path.

Distinguishing Local Damage from Fabric‑Wide Failure

When tenant‑scoped metrics deviate while global fabric indicators stay within normal bounds, the evidence points to local damage—a problem confined to the tenant’s data path (e.g., mis‑queuing, ACL drop, micro‑burst on a leaf‑to‑leaf link). Conversely, concurrent deviations in both tenant‑scoped and fabric‑wide metrics suggest a fabric‑wide failure.

By leveraging tenant‑scoped telemetry, correlating timelines, and interpreting the signals above, operators can quickly isolate partial tenant degradation and avoid lengthy, misguided fabric‑centric troubleshooting.