Flattening hierarchical paths without label-cardinality suicide

Introduction

gNMI streams model‑driven telemetry as leaf values identified by a full YANG‑modeled path (e.g., /interfaces/interface[name=GigabitEthernet0/0/0]/counters/in-octets). Converting these leaves into time‑series metrics requires translating the path into a metric name and a set of labels while preserving enough context for debugging, automation, and correlation—without causing label‑cardinality explosion or hiding subtree identity in opaque strings.

Design Considerations

Balancing Context and Label Cardinality

Each distinct label combination creates a new time series. Exporting every key as a separate label can quickly generate millions of series in large networks, stressing Prometheus ingestion limits, memory, and query performance. Conversely, discarding keys reduces diagnostic granularity and forces fragile string matching on metric names or opaque labels.

Subtree Identity Preservation

A gNMI path encodes not only the leaf but also the containing subtree (e.g., the /interfaces/interface list). Preserving subtree identity enables aggregation (summing counters across a line‑card), filtering (selecting interfaces in a VRF), and schema evolution (adding new containers without breaking existing metrics). If subtree identity is lost, downstream consumers must re‑parse device‑specific naming conventions.

Avoiding Opaque Strings

Serializing the entire path into a single label (e.g., gnmi_path="…") preserves information but creates opaque strings that prevent label‑based arithmetic, hinder efficient indexing, and complicate alerting and dashboarding. A robust schema exposes meaningful path components as structured labels while keeping the total label set bounded.

Path‑to‑Metric Schema Design

Hierarchical Labeling Approach

We use a tiered labeling model that mirrors the YANG hierarchy but caps dynamic label depth at a configurable threshold D (typically 2–3). The model consists of:

1. Metric Name – Derived from the leaf identifier (e.g., if_in_octets, bgp_peer_state). Stable across devices and vendors.
2. Fixed Labels – Small set of high‑value identifiers always exported:
   • device – unique device identifier (serial number or sysName)
   • vendor – optional, for cross‑vendor correlation
   • model – optional, for hardware‑specific baselines
3. Dynamic Labels – One label per keyed list encountered in the path, up to depth D. Each label uses the YANG leaf name of the key (e.g., interface_name, vrf_name). If the path exceeds D, the remaining hierarchy is collapsed into a single gnmi_path_suffix label containing the rest of the path as a percent‑encoded string (still queryable but limited in cardinality).

Example (D=2)

Operators can increase D for critical subtrees (e.g., BGP neighbors) while keeping it low for high‑branching containers like interfaces.

gNMI Leaf Context Preservation Techniques

• Key Extraction – Parse each list entry and expose its key as a label, using the YANG leaf name from the device’s capabilities or a local YANG cache for semantic consistency.
• Value Normalization – Convert key values to a canonical form (lower‑case interface names, strip leading zeros from VLAN IDs) to reduce cardinality from formatting differences.
• Timestamp Alignment – Preserve the gNMI timestamp as the metric timestamp; store the original gNMI timestamp in a gnmi_timestamp label if the collection system applies its own timestamp, enabling latency analysis.
• Unit Annotation – Append a standardized unit suffix to the metric name (e.g., _bytes, _packets, _percent) derived from the YANG units statement, allowing automatic unit conversion in Prometheus.

Handling Subtree Identity in Labels

1. Explicit Subtree Labels – For known aggregation points (e.g., /linecard, /fabric, /vrf), create dedicated labels (linecard_name, fabric_id, vrf_name) regardless of depth, configured via a subtree whitelist that operators can update without schema changes.
2. Implicit Hierarchy via Label Ordering – Prometheus allows label‑based grouping; ordering labels from root to leaf (device → subtree → instance) enables implicit aggregation using by (device, linecard_name). The ordering is enforced in the label generation code.

If a subtree is not whitelisted and exceeds the depth limit, its identity remains in the gnmi_path_suffix label, which can still be filtered with regex but at higher query cost.

Implementing the Path‑to‑Metric Schema

Example Code for Label Generation (Go)

package gnmi2metric

import (
	"fmt"
	"strings"

	"github.com/openconfig/gnmi/proto/gnmi"
	"github.com/openconfig/ygot/ygot"
)

// LabelGenConfig holds tunable parameters.
type LabelGenConfig struct {
	MaxDepth        int               // D: maximum number of dynamic key labels
	SubtreeWhitelist map[string]bool   // e.g., map["/linecard"]=true
	DeviceID        string            // static device identifier
	Vendor          string
	Model           string
}

// MetricLabelSet returns the metric name and map of labels for a gNMI Update.
func MetricLabelSet(update *gnmi.Update, cfg *LabelGenConfig) (string, map[string]string) {
	// 1. Derive metric name from leaf path and YANG type.
	metricName := leafToMetricName(update.Path) // e.g., if_in_octets
	// 2. Initialize label map with static fields.
	labels := map[string]string{
		"device":   cfg.DeviceID,
		"vendor":   cfg.Vendor,
		"model":    cfg.Model,
	}
	// 3. Walk the path elements, extracting keys.
	dynDepth := 0
	for _, elem := range update.Path.GetElem() {
		// Stop adding dynamic labels once depth limit is reached.
		if dynDepth >= cfg.MaxDepth {
			suffix := pathToString(update.Path)
			labels["gnmi_path_suffix"] = suffix
			break
		}
		// If this element is a keyed list, expose its key(s).
		if len(elem.GetKey()) > 0 {
			for k, v := range elem.GetKey() {
				labelName := fmt.Sprintf("%s_name", snakeCase(k))
				labels[labelName] = normalizeValue(v)
				dynDepth++
			}
		}
		// If this element matches a whitelisted subtree, add a dedicated label.
		fullPath := pathToString(update.Path[:len(update.Path)-len(elem.GetElem())+1]) // pseudo‑code
		if cfg.SubtreeWhitelist[fullPath] {
			labelName := fmt.Sprintf("%s_name", snakeCase(elem.GetName()))
			// Simplified lookup; real implementation would map the key to its value.
			labels[labelName] = normalizeValue(elem.GetKey()[elem.GetName()])
		}
	}
	return metricName, labels
}

// Helper functions omitted for brevity: leafToMetricName, pathToString, snakeCase, normalizeValue.

The function respects the depth limit, adds static labels, and optionally injects whitelisted subtree labels. It can be integrated into a gNMI collector (e.g., OpenTelemetry Collector’s gnmi receiver) or a sidecar that translates updates to Prometheus remote‑write.

CLI Examples for Schema Deployment

OpenTelemetry Collector configuration (YAML)

receivers:
  gnmi:
    endpoint: router01.example.com:57400
    username: telemetry
    password: ${GNMI_PASSWORD}
    tls:
      insecure_skip_verify: false
    subscription:
      - path: /interfaces/interface/state/counters
        mode: stream
        sample_interval: 10s
      - path: /network-instances/network-instance/state/bgp/neighbors/neighbor/state
        mode: stream
        sample_interval: 30s

processors:
  gnmi2metric:
    max_depth: 2
    subtree_whitelist:
      - /linecard
      - /fabric
    device_id: "{{ .System.Hostname }}"
    vendor: "acme"
    model: "router-9000"

exporters:
  prometheusremotewrite:
    endpoint: http://prometheus.example.com:9090/api/v1/write
    # Optional: basic auth, TLS, etc.

service:
  pipelines:
    metrics:
      receivers: [gnmi]
      processors: [gnmi2metric]
      exporters: [prometheusremotewrite]

Using gnmic to verify label generation

# Subscribe to a single leaf and print the JSON payload.
gnmic -a router01.example.com:57400 -u telemetry -p $GNMI_PASSWORD \
    --tls skip-verify \
    subscribe --path /interfaces/interface[name=GigabitEthernet0/0/0]/state/counters/in-octets \
    --mode once --encoding json

# Expected output (simplified):
# {
#   "update": [
#     {
#       "path": "/interfaces/interface[name=GigabitEthernet0/0/0]/state/counters/in-octets",
#       "val": {"uint64": 12345678}
#     }
#   ],
#   "timestamp": 1727845200000000000
# }

The collector transforms this into a Prometheus series:

if_in_octets{device="router01",interface_name="GigabitEthernet0/0/0"} 12345678 1727845200

Integration with Existing Monitoring Tools

• Prometheus – Direct remote‑write or via the Prometheus OpenTelemetry Adapter. Metric names follow the Prometheus convention ([a-zA-Z_:][a-zA-Z0-9_:]*); labels are fully compatible with Prometheus’ data model.
• Grafana – Dashboards can use standard by (device, interface_name) grouping. The gnmi_path_suffix label remains searchable but should be avoided in high‑cardinality panels.
• Alertmanager – Alert rules can reference any label; example high‑interface‑utilization alert:

  - alert: HighInterfaceUtilization
      expr: (rate(if_in_octets[5m]) + rate(if_out_octets[5m])) / if_speed > 0.9
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Interface {{ $labels.interface_name }} on {{ $labels.device }} is >90% utilized"
        description: "Utilization measured over the last 5 minutes."

• Logging & Tracing Correlation – The device label can join with log streams carrying the same identifier (e.g., Loki’s pipeline_stages). For traces, inject the device label as a resource attribute in OpenTelemetry instrumentation.

Troubleshooting the Path‑to‑Metric Schema

Common Issues with Label Cardinality

Mitigation Strategies

• Tune MaxDepth per subnet of telemetry (interfaces vs. routing protocols) using separate collector instances or processor configurations.
• Leverage the subtree whitelist for known aggregation points; update it without redeploying the schema.
• Apply value normalization consistently across collectors to avoid artificial cardinality spikes.
• Monitor label cardinality with Prometheus built‑in metrics (scrape_series_sampled_limit_* and prometheus_tsdb_head_series) and alert on abnormal growth.

By following this schema, operators retain the contextual richness of gNMI leaves for effective debugging and automation while keeping label cardinality within manageable bounds.