Comparing Generated Subscription Paths with Collector Leaves: Exposing Silent Schema Gaps

Introduction to Observability Completeness

Observability completeness is the degree to which the set of telemetry signals collected from a system matches the set of signals required to answer all operational questions of interest without relying on inference, correlation, or external knowledge that is not directly observable. Let:

• S_intended = the set of telemetry paths (subscription paths) that automation declares it will request, derived from the desired intent.
• S_rendered = the set of paths actually emitted by the telemetry agent after applying local configuration, filters, and any model-driven transformations.
• S_applied = the set of paths that reach the collector after transport and any intermediate processing.
• S_observed = the set of leaves (individual data points) that the collector stores, indexes, and makes available for querying. Observability completeness holds when S_intended ⊆ S_observed and every element of S_intended can be uniquely mapped to a leaf in S_observed with preserved semantics.

Understanding Subscription Paths and Collector Leaves

Overview of Subscription Paths

A subscription path is a YANG-modeled XPath that identifies a subtree of data to be streamed. The automation may also apply path normalization: collapsing duplicate leaf-lists, adding key predicates, or inserting namespace prefixes.

Collector Leaves and Their Role in Observability

A collector leaf is the atomic telemetry datum that arrives at the observability backend after transport, decoding, and optional transformation. The collector’s leaf set (S_observed) is a function of the submitted paths, transport fidelity, collector parsing rules, and post-processing.

Silent Schema Gaps and Their Impact

Definition and Explanation of Silent Schema Gaps

A silent schema gap occurs when the automation believes a subscription path P is active and valid, the telemetry agent successfully renders P, but the collector receives a stream for P without the corresponding leaf(s) in S_observed due to a YANG schema mismatch. Typical causes include:

Troubleshooting Silent Schema Gaps

Identifying Silent Schema Gaps

A systematic approach:

1. Collect the four sets (intended, rendered, applied, observed).
2. Normalize each path to a canonical form.
3. Compute set differences.
4. Validate schema compatibility for each element in Missing.

Tools and Techniques for Detecting Schema Gaps

Code Examples for Troubleshooting Silent Schema Gaps

CLI Commands for Schema Gap Detection

# 1. Export intended paths from Git
git show HEAD:config/intent/subscriptions.yaml > intended_paths.txt
# 2. Render paths on the telemetry agent
gnmi_cli -addr telemetry-agent:9339 -debug -subscribe -paths intended_paths.txt > rendered_paths.txt 2>&1
# 3. Capture raw gNMI messages on the wire
tcpdump -i any port 9339 -w gnmi.pcap -c 1000
# 4. Extract observed metric names from Prometheus
curl -s http://prometheus:9090/api/v1/label/__name__/values | jq -r '.data[]' > observed_metrics.txt

Programming Language Examples for Schema Gap Identification

Python (set-based diff + YANG validation)

import subprocess, json, sys
from lxml import etree

def run(cmd):
    return subprocess.check_output(cmd, shell=True, text=True).strip()

# 1. Load intended paths
intended = set(run("cat intended_paths.txt").splitlines())
# 2. Load observed metric names from Prometheus
observed = set(run("curl -s http://prometheus:9090/api/v1/label/__name__/values | jq -r '.data[]'").splitlines())