Build a Compact Telemetry Playbook

Revealing cross‑zone asymmetry from histogram buckets, per‑zone error ratios, and request‑duration quantiles.

Introduction to Telemetry Playbook

Overview of Telemetry Data

These signals are scraped from each zone’s Prometheus federation target and stored in a central TSDB.

Example scrape payload

# HELP request_duration_seconds_bucket Request latency histogram
# TYPE request_duration_seconds_bucket histogram
request_duration_seconds_bucket{le="0.005",zone="us-east-1",service="frontend"} 12
request_duration_seconds_bucket{le="0.01",zone="us-east-1",service="frontend"} 57
request_duration_seconds_bucket{le="0.025",zone="us-east-1",service="frontend"} 210
...

The le label defines the bucket upper bound; the metric value is the cumulative count of observations ≤ that bound. Zone‑level granularity is present via the zone label.

Why Zone‑Level Analysis Matters

Cluster‑wide averages hide latency spikes that affect only a subset of zones (e.g., a mis‑configured NIC in us-west-2). Keeping the zone label lets us compute per‑zone ratios and quantiles, exposing asymmetry that would be diluted in a global average.

Operational question
Is request latency or error rate significantly worse in any single zone compared to the fleet average, and does this asymmetry correlate with observable infrastructure events?

Answering this requires:

1. Histogram‑derived quantiles per zone.
2. Per‑zone error‑to‑request ratios.
3. Correlation with control‑plane events (Kubernetes node conditions, BGP flaps, etc.).

If any signal lacks the zone label, note the blind spot explicitly.

Designing the Telemetry Playbook

Histogram Buckets for Request Duration

A histogram approximates any quantile without storing every observation. Define a standard bucket set aligned with SLOs:

# prometheus.yml snippet
scrape_configs:
  - job_name: 'frontend'
    metrics_path: /metrics
    static_configs:
      - targets: ['frontend-us-east-1:9090','frontend-us-west-2:9090']
    relabel_configs:
      - source_labels: [__address__]
        target_label: zone
        regex: '(.*)-(us-east-1|us-west-2):9090'
        replacement: '$2'

Resulting series:

request_duration_seconds_bucket{le="0.1",zone="us-east-1"}
request_duration_seconds_bucket{le="0.1",zone="us-west-2"}
...

Notice: The le label values must be identical across zones; otherwise quantile comparison is invalid.

Per‑Zone Error Ratios for Fault Detection

Error ratio per zone:

error_ratio_zone = sum(rate(requests_total{code=~"5..",zone="$ZONE"}[5m]))
                   /
                   sum(rate(requests_total{zone="$ZONE"}[5m]))

Expose it as a recording rule to avoid on‑the‑fly division:

# recording_rules.yml
groups:
  - name: zone_error_ratio
    rules:
      - record: zone:error_ratio:5m
        expr: |
          sum by (zone) (rate(requests_total{code=~"5.."}[5m]))
          /
          sum by (zone) (rate(requests_total[5m]))

Notice: The rule uses sum by (zone) to preserve granularity. If requests_total lacks a zone label, the rule yields a single global series—a blind spot that must be fixed by re‑instrumenting the client library.

Request‑Duration Quantiles for Performance Analysis

From the histogram compute the 95th‑percentile latency per zone:

# 95th‑pct latency per zone over 5 min
histogram_quantile(
  0.95,
  sum by (le, zone) (rate(request_duration_seconds_bucket[5m]))
)

Result example:

{zone="us-east-1"} 0.042
{zone="us-west-2"} 0.128

Notice: A spread > 2× between zones signals asymmetry. The function assumes linear interpolation within the bucket containing the quantile; if the final bucket is +Inf, the estimate becomes unreliable—keep this sampling limit in mind.

Implementing the Telemetry Playbook

Data Collection and Storage

1. Instrumentation – Sidecar proxies emit Prometheus metrics with a zone label via downward API or environment variable.
2. Scrape – Prometheus servers in each zone scrape locally; a central Prometheus federates zone:error_ratio:5m and the histogram aggregates.
3. Remote Write – All samples are remotely written to a Cortex/Tempo cluster for long‑term storage (2 y retention).

Missing signal: If the sidecar does not export zone, add it via the pod annotation prometheus.io/label.

Data Processing and Aggregation

Recording rule for histogram sums (reduces query load):

groups:
  - name: zone_histogram_agg
    rules:
      - record: zone:request_duration_seconds_bucket_sum:5m
        expr: sum by (le, zone) (rate(request_duration_seconds_bucket[5m]))

Notice: The rule preserves the le label so histogram_quantile can still be applied later.

Example Code for Data Analysis

Python snippet that flags zones whose 95th‑pct latency exceeds 2× the median:

import os, requests, statistics

PROM = os.getenv("PROM_URL", "http://prometheus:9090")
QUERY = """
histogram_quantile(
  0.95,
  sum by (le, zone) (rate(request_duration_seconds_bucket[5m]))
)
"""

def fetch():
    r = requests.get(f"{PROM}/api/v1/query", params={"query": QUERY})
    return r.json()["data"]["result"]

def analyze():
    series = fetch()
    vals = [float(s["value"][1]) for s in series]
    med = statistics.median(vals)
    outliers = [s for s, v in zip(series, vals) if v > 2 * med]
    for s in outliers:
        zone = s["metric"]["zone"]
        latency = s["value"][1]
        print(f"⚠️ Zone {zone} 95th‑pct latency {latency}s > 2× median ({med}s)")

if __name__ == "__main__":
    analyze()

Notice: Using the median provides a robust baseline; sparse series (e.g., due to scrape failures) can skew the median—note this sampling limit.

Troubleshooting Cross‑Zone Asymmetry

Identifying Asymmetry with Histograms

1. Plot per‑zone 95th‑pct latency over time (Grafana heatmap).
2. Look for zones where the line diverges > 50 ms from the fleet median.

Example PromQL result

# Query: histogram_quantile(0.95, sum by (le, zone) (rate(request_duration_seconds_bucket[5m])))
# Time: 2025-09-24T12:00:00Z
{zone="us-east-1"} 0.038
{zone="us-west-2"} 0.112
{zone="eu-central-1"} 0.041

Notice: us-west-2 shows ~3× higher latency.

Correlating with Envoy logs

[2025-09-24 12:00:12] upstream_rq_timeout: 12ms (max: 10ms) cluster=frontend_outbound
[2025-09-24 12:00:13] envoy.filters.network.tcp_proxy: upstream connect failed: Connection refused

Notice: Timeout spikes and connection refusals coincide with the latency rise, suggesting backend overload or a mis‑configured circuit breaker.

Analyzing Error Ratios for Faulty Zones

Compute the difference between a zone’s error ratio and the global average:

zone:error_ratio:5m - avg without _zone (zone:error_ratio:5m)

Sample output

{zone="us-west-2"} 0.018   # 1.8% error ratio
{zone="us-east-1"} 0.002
{zone="eu-central-1"} 0.001

Notice: us-west-2 error ratio is an order of magnitude higher.

Correlating with control‑plane events

kubectl get nodes -l topology.kubernetes.io/zone=us-west-2 -o wide

Output (excerpt):

ip-10-0-4-5.us-west-2    Ready    <none>   210d  v1.28.3
ip-10-0-4-6.us-west-2    NotReady <none>   210d  v1.28.3   # <-- NodeNotReady

Notice: One node in us-west-2 is NotReady, reducing capacity and causing retries → higher error ratio.

Using Quantiles to Detect Performance Issues

Examine the spread between 50th and 99th percentiles:

# 99th‑pct latency per zone
histogram_quantile(0.99, sum by (le, zone) (rate(request_duration_seconds_bucket[5m])))

If the 99th‑pct spikes while the 50th stays flat, tail latency is worsening—often a sign of GC pauses or lock contention.

Example

{zone="us-west-2"} 0.42   # 99th‑pct
{zone="us-west-2"} 0.07   # 50th‑pct (from separate query)

Notice: The 99th‑pct is 6× the median, indicating a heavy tail likely caused by occasional slow DB queries in that zone.

Code Examples for Telemetry Analysis

CLI Commands for Data Retrieval

Fetch raw histogram buckets for a zone:

curl -g "http://prometheus:9090/api/v1/query?query=request_duration_seconds_bucket{le=\"0.1\",zone=\"us-west-2\"}"

Notice: The response includes __name__, le, zone, and the cumulative count. If the zone label is missing, the query returns a single series—a blind spot indicating missing instrumentation.

Python Code for Data Processing and Visualization

Per‑zone latency heatmap using pandas and matplotlib:

import pandas as pd, matplotlib.pyplot as plt, requests, numpy as np

def get_series(metric):
    r = requests.get(
        "http://prometheus:9090/api/v1/query_range",
        params={"query": metric, "start": "-1h", "end": "now", "step": "30s"},
    )
    return r.json()["data"]["result"]

hist = get_series("request_duration_seconds_bucket")
records = []
for s in hist:
    zone = s["metric"]["zone"]
    le   = float(s["metric"]["le"])
    for ts, val in s["values"]:
        records.append(
            {
                "time": pd.to_datetime(float(ts), unit="s"),
                "zone": zone,
                "le": le,
                "val": int(val),
            }
        )
df = pd.DataFrame(records).pivot_table(
    index="time", columns=["zone", "le"], values="val", aggfunc="sum"
)

def quantile(row, q=0.95):
    cum = row.values
    total = cum[-1]
    target = q * total
    idx = np.searchsorted(cum, target, side="left")
    if idx == len(cum):
        return np.inf
    low = row.index[idx - 1] if idx > 0 else 0
    high = row.index[idx]
    if cum[idx] == cum[idx - 1]:
        return high
    return low + (high - low) * (target - cum[idx - 1]) / (cum[idx] - cum[idx - 1])

q95 = df.apply(quantile, axis=1)
q95.unstack(level=0).plot(kind="line", title="95th‑pct latency per zone")
plt.ylabel("seconds")
plt.show()

Notice: The script reconstructs the cumulative histogram per scrape, then computes the 95th‑pct via interpolation. If the scrape interval is too coarse (e.g., 5 min), rapid spikes may be missed—note this sampling limit.

Example Queries That Hide Zone Asymmetry

A misleading global 95th‑pct that averages across zones:

# Misleading: global 95th‑pct (hides per‑zone spread)
histogram_quantile(0.95, sum by (le) (rate(request_duration_seconds_bucket[5m])))

Notice: This query drops the zone label, erasing any cross‑zone differences. Always retain zone (or other relevant labels) when comparing quantiles across failure domains.