Collector fan-out or direct device subscriptions

Introduction to gNMI Design Review

Overview of gNMI

gNMI (gRPC Network Management Interface) is a protobuf‑based, gRPC‑transport protocol defined by the OpenConfig working group for reading, writing, and subscribing to operational and configuration data on network devices. It defines three primary RPCs: Get, Set, and Subscribe. The Subscribe RPC supports stream, poll, and once modes, allowing a client to receive a continuous stream of updates (telemetry) or a snapshot of state. gNMI uses TLS for transport security and can carry authentication tokens (e.g., JWT) or username/password credentials via gRPC metadata.

Importance of gNMI in Network Management

Modern network automation relies on a standardized, model‑driven interface to decouple orchestration logic from vendor‑specific CLIs. gNMI provides:

• Vendor‑neutral data models (OpenConfig, IETF, or vendor extensions) expressed in YANG.
• Efficient push‑based telemetry reducing polling overhead.
• Transactional configuration via Set with explicit error reporting.
• Fine‑grained access control through gRPC authentication and authorization mechanisms.

When scaling to hundreds or thousands of devices, the placement of the gNMI termination point—whether each downstream consumer talks directly to the device or a shared termination layer fronts the devices—has profound effects on isolation, replay capability, backpressure handling, and credential management.

Design Options for gNMI Implementation

Terminating gNMI Once Per Device (Centralized Proxy)

Benefits

• Single source of truth for credentials – the proxy holds device‑side username/password or token, shielding downstream systems from secret exposure.
• Connection pooling and reuse – a single long‑lived gNMI session per device serves multiple subscribers, reducing TLS handshake overhead and device CPU load.
• Built‑in replay and buffering – the proxy can maintain a local cache of recent updates (e.g., a ring buffer) and replay them to new subscribers without re‑querying the device.
• Backpressure isolation – slow or misbehaving subscribers affect only the proxy’s outbound queues; the device‑side gNMI stream remains unaffected.
• Simplified policy enforcement – TLS mutual authentication, RBAC, and audit logging can be enforced at the proxy layer once per device.

Drawbacks

• Additional hop and potential bottleneck – the proxy introduces latency (typically 0.5‑2 ms per hop) and becomes a failure domain; if the proxy crashes, all subscribers lose visibility.
• State replication complexity – to provide accurate replay, the proxy must mirror device state, which can diverge if the device pushes updates faster than the proxy can ingest.
• Credential centralization risk – compromising the proxy yields access to all managed devices; requires hardened storage (e.g., HashiCorp Vault, AWS KMS) and strict access controls.
• Limited visibility into device‑side gNMI errors – some devices return detailed error codes in the gNMI Status; the proxy may genericize them, complicating root‑cause analysis.

Letting Each Downstream System Subscribe Directly (Distributed Subscriptions)

Benefits

• Zero‑hop latency – subscribers receive updates straight from the device, eliminating proxy‑induced delay.
• Failure isolation per subscriber – a misbehaving subscriber only impacts its own TCP/gRPC stream; other subscribers and the device remain unaffected.
• No central credential store – each subsystem can manage its own secrets, reducing the blast radius of a credential leak.
• Straightforward scaling – adding a new consumer merely requires establishing another gNMI session; no need to resize a shared proxy tier.

Drawbacks

• Credential proliferation – each subsystem must retain device credentials, increasing the attack surface and complicating secret rotation.
• Duplicate device load – each subscription incurs a separate gNMI stream, leading to linear CPU and memory growth on the device proportional to the number of subscribers.
• Limited replay capability – devices typically do not retain historical telemetry; a new subscriber only sees updates from the moment of subscription unless the device implements its own buffering (rare).
• Backpressure propagation – a slow subscriber applies backpressure directly on the device’s gNMI server, potentially throttling updates for all other subscribers and impacting device control‑plane performance.
• Operational overhead – monitoring, TLS certificate management, and version compatibility must be repeated for each consumer.

Trade‑Offs in gNMI Design

Isolation and Security Considerations

Impact on Isolation

• Centralized termination creates a trust boundary at the proxy: all downstream systems share the same TLS session to the device. A compromised downstream client cannot directly inject malicious gNMI Set requests unless the proxy forwards them; however, a compromised proxy can impersonate any client.
• Distributed subscriptions place the trust boundary at each consumer. Isolation is stronger in the sense that a breach of one consumer does not automatically grant access to others, but the device itself sees many independent sessions, increasing the attack surface on its gNMI listener.

Security Implications

Replay and Backpressure Mechanisms

Handling Replay

• Centralized – The proxy can implement a replay buffer (in‑memory ring buffer or persistent log) that stores the last N updates per path. When a new subscriber connects, the proxy first streams buffered events, then switches to the live stream. Buffer size must be tuned: too small loses history; too large consumes proxy memory.
• Distributed – Relies on device‑side replay, which is uncommon. Some high‑end platforms support gNMI server‑side caching (e.g., Juniper’s telemetry buffer), but most devices drop old updates. Consequently, new subscribers see only live data, creating a potential observability gap during subscriber restart or scaling events.

Managing Backpressure

• Centralized – The proxy applies flow‑control at the outbound side (toward subscribers) using gRPC’s built‑in windowing. If a subscriber is slow, the proxy buffers or drops messages based on a configurable policy (e.g., drop oldest, drop newest, or apply backpressure to the inbound device stream). The inbound device stream remains unaffected unless the proxy’s internal buffers overflow, at which point the proxy can send a ResourceExhausted status to the device, causing the device to apply its own backpressure.
• Distributed – Backpressure propagates directly to the device. A slow subscriber reduces the advertised gRPC receive window, causing the device’s gNMI server to pause sending. If multiple subscribers are slow, the device may experience head‑of‑line blocking, delaying updates for fast subscribers. Devices typically lack sophisticated QoS per‑subscription, so the effect is blunt.

Credentials and Authentication Management

Centralized gNMI

• Secrets are stored in a secret‑management system (Vault, AWS Secrets Manager, Azure Key Vault) and retrieved at proxy startup or via short‑lived tokens (e.g., Vault dynamic secrets). The proxy presents a single set of credentials to each device, simplifying rotation: update the secret in the vault, signal the proxy to reload, and all downstream consumers automatically use the new credential without modification.
• Access control: downstream systems authenticate to the proxy via mutual TLS or JWT; the proxy maps the client identity to internal roles (e.g., telemetry-reader, config-writer). This enables credential delegation without exposing device secrets.

Distributed gNMI

• Each subsystem must obtain and store device credentials. Rotation requires a coordinated rollout: update the secret in the vault, then restart or signal each consumer to refresh its credentials. Automation (e.g., sidecar injector, init container) can reduce manual effort but adds complexity.
• Authentication: mTLS is common; each consumer needs a client certificate signed by a CA trusted by the device. Managing many CAs or many device‑specific trust stores becomes operationally heavy at scale.
• Authorization: relies on device‑provided ACLs (if any). Many devices only support a simple username/password check, offering no per‑path or per‑operation granularity.

Troubleshooting gNMI Implementations

Common Issues in gNMI Termination

Debugging gNMI Connection Issues

1. TLS handshake failures – verify proxy’s client certificate matches the device’s expected SAN; check device logs for unknown_ca or certificate_unknown.
2. Authentication rejections – ensure the proxy retrieves the correct secret; test with gnmic -a <device>:<port> -u <user> -p <pass> capabilities.
3. gNMI service not responding – confirm the device’s gNMI server is enabled (show running-config | include gnmi) and listening on the correct TCP port (default 9339).
4. Proxy crashes – inspect proxy stdout/stderr for panics; enable verbose logging (-log-level=debug) to see RPC errors.

Resolving gNMI Subscription Conflicts

• Duplicate path subscriptions – if the proxy allows multiple subscribers on the exact same path, it may multiplex updates; conflicts arise when one subscriber requests mode: stream and another mode: poll. Resolve by configuring the proxy to normalize modes (e.g., always upgrade to stream) or reject incompatible subscriptions with a clear FailedPrecondition status.
• Buffer overruns – monitor proxy internal queue length; if consistently high, increase buffer size or apply a drop policy to avoid losing updates for all subscribers.
• Credential mismatch after rotation – verify that the proxy has reloaded secrets; send a SIGHUP or use a sidecar that watches the secret volume.

Troubleshooting Distributed gNMI Subscriptions

Identifying Issues in Downstream Subscriptions

• Subscription storms – use device‑side gNMI metrics (if exposed) to detect sudden spikes in active streams; correlate with consumer deployment events.
• Intermittent disconnects – check consumer logs for context canceled or resource exhausted gRPC status; verify network stability (TCP retransmits, MTU mismatches).
• Missing telemetry – confirm the consumer’s subscription paths are correctly formatted (e.g., using the proper YANG model prefix) and that the device actually publishes those paths.

Resolving Authentication and Authorization Issues

• mTLS handshake failures – ensure the consumer’s client certificate is within validity period and signed by a CA the device trusts; use openssl s_client -connect <device>:<port> -cert <cert> -key <key> to debug.
• Username/password rejections – verify credential store synchronization; some devices lock accounts after failed attempts.
• Authorization errors (e.g., PERMISSION_DENIED) – check device ACLs; if the device lacks fine‑grain control, consider moving to a proxy model for policy enforcement.

Code and CLI Examples for gNMI

Configuring gNMI Termination using CLI

Example CLI Commands for gNMI Setup
Assuming a proxy implementation based on the open‑source gnmi_proxy (Go) that terminates gNMI per device and exposes a local gRPC endpoint for consumers:

# 1. Start Vault agent to fetch dynamic secrets
vault agent -config=/etc/vault/agent.hcl &
# Agent writes device credentials to /run/secrets/gnmi/<device>

# 2. Launch the proxy for a specific device
gnmi_proxy \
  --device-addr router01.example.com:9339 \
  --device-cert /run/secrets/gnmi/router01/tls.crt \
  --device-key  /run/secrets/gnmi/router01/tls.key \
  --device-ca   /run/secrets/gnmi/router01/ca.crt \
  --listen-addr 0.0.0.0:10000 \
  --client-cert /run/secrets/proxy/tls.crt \
  --client-key  /run/secrets/proxy/tls.key \
  --client-ca   /run/secrets/proxy/ca.crt \
  --replay-buffer-size 10000 \
  --backpressure-policy drop-newest

The proxy opens a gNMI session to the device (--device-*) and exposes a local gNMI endpoint (--listen-addr) for downstream systems. Mutual TLS is enforced both ways.

Verifying gNMI Connection Status

# Check proxy health endpoint (if exposed)
curl -k https://localhost:10001/healthz
# Expected: {"status":"ok","device":"router01.example.com","subscribers":3}

# Use gnmic to query capabilities via the proxy
gnmic -a localhost:10000 \
      --insecure \
      capabilities

Implementing gNMI Subscriptions using Code

Python (using gnmi_client library)

import grpc
from gnmi import gnmi_pb2, gnmi_pb2_grpc
from google.protobuf.any_pb2 import Any

def subscribe(device_addr, token, paths):
    # Load TLS credentials
    creds = grpc.ssl_channel_credentials(
        root_certificates=open('ca.pem', 'rb').read(),
        private_key=open('client.key', 'rb').read(),
        certificate_chain=open('client.crt', 'rb').read()
    )
    # Add bearer token as metadata
    call_creds = grpc.metadata_call_credentials(
        lambda context, callback: callback([('authorization', f'Bearer {token}')], None)
    )
    composite_creds = grpc.composite_channel_creds(creds, call_creds)

    channel = grpc.secure_channel(device_addr, composite_creds)
    stub = gnmi_pb2_grpc.gNMIStub(channel)

    # Build subscription request (example: stream mode)
    subscribe = gnmi_pb2.SubscribeRequest(
        subscribe=gnmi_pb2.SubscriptionList(
            subscription=[
                gnmi_pb2.Subscription(path=gnmi_pb2.Path(elem=[gnmi_pb2.PathElem(name=p)]),
                                      mode=gnmi_pb2.SubscriptionMode.STREAM,
                                      sample_interval=10_000_000_000)  # 10 seconds
                for p in paths
            ],
            mode=gnmi_pb2.SubscriptionList.STREAM
        )
    )

    for resp in stub.Subscribe(iter([subscribe])):
        print(resp)

This snippet establishes a mutual‑TLS channel, injects a JWT token via gRPC metadata, and opens a streaming subscription for the supplied YANG paths.

End of review.