Honest 95th percentile capacity in PromQL

Introduction to PromQL and Network Capacity Observation

PromQL is the query language used by Prometheus to select and aggregate time‑series data. It works on raw samples collected via scrape intervals (typically 15 s–1 min) and supports instant vectors, range vectors, and functions such as rate, increase, quantile_over_time, and histogram_quantile. PromQL expressions are evaluated against the current time unless a range modifier ([range]) creates a range vector, enabling windowed calculations like percentiles over a period.

Network capacity observation answers the operational question: “What is the sustained bandwidth usage that the network must provision for, ignoring short‑lived bursts?” In service‑provider and data‑center environments, capacity planning, burstable billing, and SLA verification rely on a statistic that reflects typical load while discounting the top N % of samples. The 95th‑percentile is the de‑facto standard because it approximates the “usable” capacity after discarding the highest 5 % of measured intervals, which are often caused by transient spikes, measurement noise, or scheduled bulk transfers.

Understanding 95th‑Percentile Network Capacity Observation

Definition and Significance

Given n measured bandwidth samples (e.g., bits per second per interface over each scrape interval), the 95th‑percentile is the value x such that 95 % of the samples are ≤ x and 5 % are > x. In billing, the top 5 % are ignored, allowing customers to burst without penalty while still charging for the sustained usage that drives infrastructure cost.

Statistically, the 95th‑percentile is a robust estimator of the upper bound of typical traffic; it is less sensitive to outliers than the mean and directly maps to the “committed information rate” (CIR) used in many service contracts.

Calculating 95th‑Percentile in PromQL

Prometheus provides quantile_over_time(φ, range_vector) (available since v2.27) which returns the φ‑quantile of the values in a range vector over the look‑back period. To compute the 95th‑percentile of interface bandwidth:

# Interface inbound bandwidth (bits per second) over the last 2 hours,
# 95th‑percentile calculated over 5‑minute rate samples.
quantile_over_time(
  0.95,
  rate(ifHCInOctets[5m])[2h:]
)

Explanation

• ifHCInOctets is a cumulative counter (octets received).
• rate(...[5m]) converts the counter to a per‑second rate; multiply by 8 to obtain bits per second if needed.
• The outer [2h:] creates a range vector of the instantaneous rate series over the last two hours.
• quantile_over_time(0.95, …) returns the 95th‑percentile of those rates.

For outbound or combined direction, repeat the expression for ifHCOutOctets and either sum the two directions before applying the quantile or compute each direction separately and take the max, depending on billing policy.

Building a Repeatable Method for 95th‑Percentile Observation

Step 1: Data Collection and Instrumentation

Instrumentation details

• Use the Prometheus SNMP exporter or a gNMI exporter (e.g., gnmi_exporter from the OpenConfig project) to pull ifHCInOctets/ifHCOutOctets from routers/switches.
• Ensure the exporter publishes the interface name (ifName) and optionally the interface index (ifIndex) as labels to enable per‑interface aggregation.
• Verify that counters are monotonic; if wraps are observed, increase scrape frequency or enable counter reset handling in the exporter.

Missing signals to note

• No direct measurement of application‑level throughput (e.g., TCP goodput) – only link‑layer octets.
• No per‑flow or per‑class‑of‑service breakdown unless additional telemetry (e.g., flow‑export, telemetry model openconfig-interfaces:counters) is added.

Step 2: Query Construction in PromQL

A reusable template for the 95th‑percentile of total (in + out) bandwidth per interface:

# 1. Compute per‑direction rates (bits/s)
rate_in  = rate(ifHCInOctets[5m]) * 8
rate_out = rate(ifHCOutOctets[5m]) * 8

# 2. Combine directions (optional: use max if billing uses peak direction)
total_rate = (rate_in + rate_out)   # sum
# total_rate = max(rate_in, rate_out)   # alternative

# 3. Apply 95th‑percentile over the desired look‑back window (e.g., 1 h)
quantile_over_time(0.95, total_rate[1h:])

Parameterization

• Scrape interval ([5m]) should be at least twice the expected burst duration to capture the shape of a spike.
• Look‑back window ([1h:]) determines the billing period; typical contracts use 1 h, 5 min, or 24 h windows. Adjust accordingly.

Step 3: Handling Bursty Traffic and Edge Cases

1. High‑resolution scraping – For links prone to sub‑second microbursts (e.g., data‑center spine links), reduce the scrape interval to 5 s or 1 s and increase the rate window accordingly (rate(...[10s])). This reduces the chance that a burst is completely missed between samples.

2. Counter reset detection – If the exporter does not automatically handle counter wraps, add a safeguard:

   increase(ifHCInOctets[5m]) > 0   # ensures monotonic increase

   Discard samples where the condition fails (using unless).

3. Interface flaps – When an interface goes down, the counter resets to zero, causing an artificial spike in rate. Filter by operational state:

   rate(ifHCInOctets[5m]) * 8 unless on() ifOperState == 0

4. Missing samples – Gaps cause rate to produce NaN. Use absent_over_time to detect gaps and optionally fill with the last known value (last_over_time) if the gap is short (< 2 × scrape interval).

5. Bulk‑transfer windows – If scheduled backups create predictable bursts, consider excluding known time windows via unless time() >= ... or by creating a recording rule that computes the 95th‑percentile on a filtered range vector ([1h:] offset by -1h to avoid the backup window).

Troubleshooting Common Issues in 95th‑Percentile Observation

Identifying and Addressing Data Gaps

• Symptom: The quantile query returns NaN or fluctuates wildly.

• Diagnosis:

  # Show scrape success per target
  up{job="snmp"}  
  # Show number of samples in the last 5 m
  count_over_time(ifHCInOctets[5m])

• If count_over_time is less than expected (e.g., < 3 samples for a 5 m window with 15 s scrape), increase scrape frequency or check network reachability/Prometheus scraper load.

Handling Missing or Inconsistent Data

• Missing counters – Some devices only export 32‑bit ifInOctets. Detect via:

  absent(ifHCInOctets) and ifInOctets  

  In such cases, fall back to the 32‑bit metric but apply a wrap‑aware counter increase function (increase) with a shorter window to reduce overflow risk.

• Inconsistent units – Verify that the exporter reports octets, not packets. Multiply by 8 only if the source is octets.

Debugging PromQL Queries

1. Instant‑vector inspection – Break the query into stages and view each intermediate series:

   # Rate series (instant vector)
   rate(ifHCInOctets[5m])  
   
   # Range vector for the last 20 m
   rate(ifHCInOctets[5m])[20m:]  

   Use the Prometheus UI “Graph” tab toggling “Show raw data” to see sample counts.

2. Quantile verification – Compare quantile_over_time against a manual histogram approximation:

   # Approximate 95th‑percentile via histogram (if you have a histogram metric)
   histogram_quantile(0.95, sum(rate(ifHCInOctets_bucket[5m])) by (le))  

   If the two diverge significantly, check that the rate window matches the histogram bucket resolution.

3. Alerting on stale data – Create a recording rule that flags when the 95th‑percentile has not updated in the last 2 × scrape interval:

   # Record the timestamp of the last successful quantile
   last_95p_timestamp = max_over_time(quantile_over_time(0.95, rate(ifHCInOctets[5m])[1h:])[5m:])  
   # Alert if now() - last_95p_timestamp > 300s  

Code Examples for 95th‑Percentile Observation in PromQL

Example 1: Simple 95th‑Percentile Query

Goal: 95th‑percentile of inbound bandwidth (Mbps) for interface eth0 on router rtr01 over the last hour.

# Inbound bits per second
rate_in_bps = rate(ifHCInOctets{device="rtr01",ifname="eth0"}[5m]) * 8

# Convert to Mbps
rate_in_mbps = rate_in_bps / 1e6

# 95th‑percentile over 1 h
quantile_over_time(0.95, rate_in_mbps[1h:])

What this proves: The value returned is the bandwidth level that 95 % of the 5‑minute rate samples fall at or below.
What it cannot prove: It does not reveal the duration or frequency of the top‑5 % spikes; a single 10‑second microburst could still affect the 95th‑percentile if the scrape interval aligns poorly.

Example 2: Handling Bursty Traffic with Increased Resolution

Goal: Capture sub‑second bursts on a 10 Gbps leaf‑spine link by scraping at 5 s and using a 10‑second rate window.

# 5 s scrape, 10 s rate to smooth jitter but retain burst sensitivity
rate_bps = rate(ifHCInOctets{device="leaf01",ifname="swp1"}[10s]) * 8
rate_gbps = rate_bps / 1e9

# 95th‑percentile over the last 10 min (more responsive to recent changes)
quantile_over_time(0.95, rate_gbps[10m:])

What this proves: With higher resolution, the 95th‑percentile reflects bursts that last multiple scrape intervals (e.g., a 200 ms microburst repeated every second will be visible).
What it cannot prove: Bursts shorter than the scrape interval (≤ 5 s) are still invisible; to capture those you would need hardware‑level telemetry (e.g., inline telemetry, sFlow, or eBPF‑based packet counters).

Example 3: Using Aggregations for Multi‑Interface Observation

Goal: 95th‑percentile of the aggregate inbound bandwidth across all interfaces in a router‑rack, excluding down interfaces.

# Rate per interface (bps)
rate_per_if = rate(ifHCInOctets[5m]) * 8

# Keep only interfaces that are up
rate_up = rate_per_if unless on() ifOperState == 0

# Sum across all interfaces (same device)
agg_bps = sum by (device) (rate_up)

# Convert to Gbps
agg_gbps = agg_bps / 1e9

# 95th‑percentile over the last 30 min
quantile_over_time(0.95, agg_gbps[30m:])

What this proves: The aggregate 95th‑percentile gives a view of the rack’s sustained load, useful for capacity planning of upstream links.
What it cannot prove: It hides per‑interface hotspots; a single saturated interface could be masked by many idle ones. To detect that, compute the 95th‑percentile per interface and then take the max across interfaces, or alert on any interface exceeding a threshold.

Billing Caveats

• Measurement window alignment – Billing periods often start at a fixed time (e.g., midnight). If your PromQL look‑back window is not aligned to the billing boundary, the 95th‑percentile may shift slightly. Use the offset modifier or schedule recording rules to match the billing cycle.
• Directional billing – Some providers bill on the max of inbound/outbound, others on the sum. Ensure your query matches the contract definition; otherwise you may over‑ or under‑charge.
• Exclusion of maintenance windows – Planned maintenance or scheduled bulk transfers can inflate the 95th‑percentile. If the contract allows exclusion, create a recording rule that filters out known time ranges before applying the quantile.
• Rounding and precision – Providers may round to the nearest Mbps or Gbps. Apply ceil or round functions in the final query if required.
• Service‑level guarantees – The 95th‑percentile does not guarantee that any single interval will stay below the billed rate; it only guarantees that 95 % of intervals do. For strict per‑interval guarantees, consider additional metrics (e.g., max‑over‑time) or traffic shaping.

Why Mean Utilization Lies

The arithmetic mean is sensitive to extreme values. A few high‑bandwidth bursts can raise the average significantly, suggesting a need for larger capacity than actually required for sustained traffic. Conversely, periods of low utilization can depress the mean, masking the need for peak capacity. The mean therefore fails to represent the usable capacity that drives infrastructure cost, whereas the 95th‑percentile discards the top 5 % of samples, providing a stable basis for burstable billing and capacity planning.

End of document.