Benchmarking Heartbeat Intervals for Stream Liveness on Quiet Devices

Introduction

A heartbeat interval is the periodic transmission of a minimal telemetry sample (timestamp + monotonic counter) from a network element to a collector to signal that the telemetry stream remains alive. Unlike data‑plane metrics, a heartbeat carries no semantic payload beyond liveness. Its two primary purposes are:

1. Liveness detection – Absence of a heartbeat within an expected window triggers a failure‑detection alarm, indicating a broken telemetry path (gNMI, NETCONF, UDP/TCP tunnel, etc.).
2. Stream health verification – Even when a device is “quiet” (no packet counters changing, no routing updates), a regular heartbeat provides a deterministic signal for timestamp correlation and drift detection.

In practice, a heartbeat is implemented as a gNMI Subscribe with a low sample_interval and ON_CHANGE suppressed, or as a periodic Get call returning a synthetic leaf such as /system/heartbeat/timestamp.

Quiet devices produce few intrinsic telemetry events. Relying solely on data‑plane metrics for liveness creates a blind spot: a missing counter update could be interpreted as either genuine low traffic or a broken telemetry session. Heartbeats close this gap by providing a source‑generated signal with a known inter‑arrival time, enabling:

• End‑to‑end path validation (isolating telemetry‑path faults from data‑plane issues).
• Correlation anchors for aligning sparse metric streams.
• Baseline for alert suppression (preventing false positives when a device is idle but the telemetry channel is healthy).

Failure‑Detection Latency

Failure‑detection latency (FDL) is the elapsed time between actual loss of telemetry connectivity and alert generation. It depends on three independent variables:

• Heartbeat interval (H) – nominal period between heartbeats.
• Missed‑sample threshold (M) – number of consecutive heartbeats that must be absent before firing an alert (commonly M = 1 for immediate detection, M ≥ 2 for hysteresis).
• Processing and propagation delay (D) – time for the collector to evaluate the missing sample, run the alert rule, and dispatch the notification (includes queueing, rule evaluation, and notification dispatch).

Mathematically, the worst‑case FDL is:

FDL_max = (M * H) + D

If M = 1, FDL_max ≈ H + D. Reducing H shortens detection windows but increases sample volume. In quiet devices, where data‑plane metrics may be emitted only every few minutes, the heartbeat often dominates telemetry volume. Therefore, H must be chosen to meet an SLA on failure detection (e.g., “detect link‑down within 5 s”) while staying within the collector’s ingest capacity.

Duplicate Samples and Ingest Cost

A heartbeat is, by definition, a duplicate sample when the device has no changing state to report. Each heartbeat adds a fixed‑size payload (typically 24–48 bytes for a gNMI Update containing a timestamp and sequence counter). With interval H, the duplicate sample rate per device is 1/H Hz. For N quiet devices, the total duplicate sample rate is N/H, influencing:

• Network bandwidth – critical for low‑speed out‑of‑band links (e.g., 9.6 kbps serial consoles).
• Collector ingest throughput – measured in samples per second (sps); many TSDBs have hard ingest limits before requiring sharding or downsampling.
• Storage write amplification – each sample creates a new timestamped block; high write rates increase compaction pressure and I/O latency.

Ingest cost extends beyond bandwidth to include CPU cycles for protobuf/unmarshaling, label indexing, timestamp alignment, memory pressure from the TSDB’s write‑ahead log and chunk buffers, and opportunity cost (heartbeats consume ingest slots that could serve higher‑cardinality metrics).

A useful monitor is the heartbeat ingest fraction:

heartbeat_fraction = (samples_from_heartbeats) / (total_ingested_samples)

If this fraction exceeds a threshold (e.g., 30 %), the system may be spending disproportionate resources on liveness checks.

Optimization Strategies

1. Adaptive heartbeat intervals – Increase H during stable periods (no data‑plane changes for a configurable window) and decrease H when volatility rises. Requires the device to expose a “stability” signal that the collector can use to adjust subscription parameters via gNMI Set.
2. Heartbeat suppression during active periods – If the device already emits frequent data‑plane samples (e.g., interface counters every second), suppress the heartbeat; the data stream itself provides liveness evidence.
3. Batch or compress heartbeats – Encode multiple heartbeats in a single gNMI Update (e.g., rolling window of timestamps) to reduce per‑packet overhead, trading a slight latency increase for lower overhead.
4. Selective metric downsampling – Apply lower retention resolution exclusively to heartbeat series (e.g., keep raw 1‑s samples for 1 h, then downsample to 1‑minute granularity) to preserve detection latency for recent events while reducing long‑term storage cost.

Managing Noisy Alerts

Noisy alerts occur when the alerting system interprets normal variability as failure. Common sources:

• Clock skew – device clock drift relative to collector inflates observed inter‑arrival time.
• Collector‑side queuing – temporary backpressure delays heartbeats beyond the alert window.
• Network jitter – variable latency on the telemetry path causes occasional missed samples without actual session loss.
• Misconfigured M threshold – M = 1 in jitter‑prone environments triggers alert storms.
• Duplicate suppression misbehavior – TSDB deduplication of identical timestamp/value can drop heartbeats that lack a changing counter, causing apparent loss.

Reduction Techniques

1. Grace period – Require M ≥ 2 or use a sliding window (e.g., “no heartbeat in the last 2 × H seconds for 3 consecutive evaluations”) to absorb occasional jitter.
2. Heartbeat timestamp delta – Alert on the difference between local collector time and the embedded heartbeat timestamp, isolating clock‑skew effects.
3. Transport‑layer correlation – Monitor underlying gNMI/TCP metrics (e.g., grpc.num_calls_started, grpc.num_calls_failed, TCP retransmits); fire alerts only when heartbeat loss coincides with transport‑layer anomalies.
4. Label filtering – Tag heartbeat series with a dedicated label (e.g., telemetry_type="heartbeat"); alert rules can treat them separately from data‑plane metrics.
5. Predictive thresholds – Use a simple moving average of observed inter‑arrival times to dynamically adjust alert thresholds, compensating for slow drift.

Example Alert Rules

PromQL (heartbeat‑only):

# Fires if no heartbeat sample observed for 2 * interval seconds
absent_over_time(heartbeat_timestamp{job="network_telemetry"}[2m])

Combined with transport health:

(absent_over_time(heartbeat_timestamp[90s]) == 1)
and
(grpc_calls_failed_total{job="network_telemetry"} > 0)

Alertmanager suppression (example):

route:
  receiver: 'telemetry-ops'
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  routes:
  - match:
      alertname: HeartbeatMissing
      severity: warning
    continue: true
  - match:
      alertname: HeartbeatMissing
      severity: critical
    receiver: 'pager'

Benchmarking Heartbeat Intervals

Methodology

A repeatable benchmark consists of four phases:

1. Baseline establishment – Deploy a known‑good telemetry stack (device → collector → TSDB → alerting) with a fixed heartbeat interval H₀ (e.g., 5 s). Record baseline metrics: ingest rate, collector CPU/memory, alert false‑positive rate, and end‑to‑end latency (via synthetic timestamps in heartbeats).
2. Parameter sweep – Vary H across a logarithmic range (e.g., 0.5 s, 1 s, 2 s, 5 s, 10 s, 30 s) while keeping all other configuration constant. For each H, run the system under a quiet‑device workload (no data‑plane changes) for a stabilization period of at least 5 × H to capture several missed‑sample windows.
3. Load injection – Add controlled background traffic (e.g., gNMI streaming of interface counters at 10 sps) to evaluate heartbeat overhead alongside active telemetry.
4. Failure injection – At a predetermined time, tear down the gNMI connection or inject network loss (using tc netem or a programmable switch) and measure observed FDL. Repeat multiple times to obtain a distribution.

Key observables per H:

• Sample rate (samples_per_second) – validates the theoretical 1/H.
• 95th‑percentile ingest latency – time from sample generation on device to availability in TSDB.
• Alert latency distribution – time from failure to alert.
• False‑positive rate – alerts raised during known‑good connectivity periods.
• Resource utilization – collector CPU, memory, network egress.

Tools and Technologies

• Device simulator – gnmi simulator (open‑source) or a custom Go program using github.com/openconfig/gnmi/client to emit heartbeats with configurable interval and sequence number.
• Traffic shaping – Linux tc with netem to add delay, jitter, and loss; useful for failure injection.
• Collector – Prometheus with prometheus-remote-write endpoint or a TSDB that accepts gNMI via an adapter (e.g., gnmi-to-prometheus).
• Metrics collection – Prometheus scraping of exporters exposing:
  • telemetry_heartbeat_samples_total
  • collector_ingest_samples_per_second
  • process_cpu_seconds_total
  • process_resident_memory_bytes
• Alerting – Alertmanager with inhibition rules to isolate heartbeat alerts.
• Orchestration – Kubernetes or bare‑metal with helm charts for consistent stack deployment across runs.
• Analysis – Python/pandas or PromQL queries exported to CSV for statistical analysis.

Example Code for Benchmarking Heartbeat Intervals

#!/usr/bin/env python3
"""
Benchmark heartbeat intervals for stream liveness.
Requires: grpcio, protobuf, prometheus_client, pandas
"""

import time
import subprocess
import pandas as pd
from prometheus_api_client import PrometheusConnect

PROM_URL = "http://prometheus:9090"
HEARTBEAT_METRIC = "telemetry_heartbeat_samples_total"
INGEST_METRIC = "collector_ingest_samples_per_second"

def run_simulator(interval_sec: float, duration_sec: int) -> subprocess.Popen:
    """
    Launch the gNMI simulator as a subprocess.
    The simulator exports a Prometheus metric at :8000/metrics.
    """
    cmd = [
        "gnmi-simulator",
        f"--heartbeat-interval={interval_sec}",
        f"--duration={duration_sec}",
        "--listen=:50051",
        "--metrics=:8000",
    ]
    return subprocess.Popen(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

def fetch_metric(prom: PrometheusConnect, metric: str, start: float, end: float) -> pd.DataFrame:
    """
    Query a Prometheus range vector and return a DataFrame with timestamp and value.
    """
    result = prom.custom_query_range(
        query=metric,
        start_time=start,
        end_time=end,
        step="15s",
    )
    # Convert to DataFrame
    df = pd.DataFrame(
        {
            "timestamp": [x[0] for x in result],
            "value": [x[1] for x in result],
        }
    )
    return df

def main():
    prom = PrometheusConnect(url=PROM_URL, disable_ssl=True)
    intervals = [0.5, 1, 2, 5, 10, 30]
    duration = 300  # 5 minutes per interval

    records = []
    for h in intervals:
        print(f"Starting benchmark with heartbeat interval {h}s")
        proc = run_simulator(h, duration)
        time.sleep(5)  # let simulator start
        start_time = time.time()
        time.sleep(duration)
        end_time = time.time()
        proc.terminate()
        proc.wait()

        # Fetch metrics
        heartbeat_df = fetch_metric(prom, HEARTBEAT_METRIC, start_time, end_time)
        ingest_df = fetch_metric(prom, INGEST_METRIC, start_time, end_time)

        # Compute simple statistics
        avg_heartbeat = heartbeat_df["value"].mean() if not heartbeat_df.empty else 0
        avg_ingest = ingest_df["value"].mean() if not ingest_df.empty else 0

        records.append(
            {
                "interval_s": h,
                "avg_heartbeat_sps": avg_heartbeat,
                "avg_ingest_sps": avg_ingest,
                "heartbeat_fraction": avg_heartbeat / avg_ingest if avg_ingest else 0,
            }
        )
        print(f"Finished interval {h}s: {records[-1]}")

    # Save results
    result_df = pd.DataFrame(records)
    result_df.to_csv("heartbeat_benchmark.csv", index=False)
    print("Benchmark complete. Results saved to heartbeat_benchmark.csv")

if __name__ == "__main__":
    main()

This script drives a gNMI simulator across a range of heartbeat intervals, collects basic Prometheus metrics, and outputs a CSV summary for further analysis. Adjust PROM_URL, metric names, and simulator command as needed for your environment.