Canarying telemetry schema changes before collector-wide rollout

Introduction to Verification‑Gated Workflows

YANG (Yet Another Next Generation) defines configuration and operational state for network devices. A leaf is the smallest scalar node; path mappings are the ways external systems translate their internal models to YANG datastore paths. Adding a leaf, renaming a path mapping, or changing normalization code can silently break three critical contracts:

1. Alerting – thresholds or notifications that depend on the leaf.
2. Storage schema – time‑series databases or data lakes expecting a specific metric name or tag.
3. Downstream automation – scripts, playbooks, or orchestrators that consume the exported data.

A verification‑gated workflow treats each change as a transaction with explicit pre‑conditions, a commit boundary, and a post‑commit verification gate. This provides observable evidence that the change does not violate any contract before it propagates to a larger fleet.

Designing the Verification‑Gated Workflow

Identifying Key Components and Dependencies

Before touching a device, enumerate the system boundary:

Create a dependency matrix (as shown) to locate where verification gates must be placed—e.g., after a YANG change but before rebuilding the normalization service.

Creating a Test Environment for Verification

A production‑like testbed should replicate:

1. YANG datastore – virtual router or containerized device (Cisco XRv, Juniper vMX, or netopeer2 with a YANG‑capable server) loading the candidate module.
2. Normalization service – exact version of the translation code pointing at the test datastore.
3. Telemetry collector – gNMI or NETCONF subscriber mirroring the production pipeline (Prometheus exporter, Kafka consumer).
4. Alerting rule set – copy of production alert rules aimed at the test telemetry.
5. Storage schema – temporary database with the same schema version as production, ready to accept the new metric/field.
6. Downstream automation harness – smoke‑test playbooks or scripts exercising the leaf via the normal consumption path.

Isolate the testbed using Docker/Kubernetes namespaces or a dedicated lab VRF. Tag all resources with a verification‑gated workflow ID (e.g., vgw-2025-09-26-01) for easy cleanup.

Integrating Verification Tools and Scripts

Select tools invokable from the CI pipeline:

• YANG validation – pyang, yanglint, or vendor SDK for syntactic/semantic checks and schema‑tree generation.
• Schema diff – yangdiff or custom script highlighting added/removed/renamed leaves and changes to type or units.
• Unit tests for normalization – pytest (Python) or table‑driven Go tests feeding sample external payloads and asserting resulting YANG nodes.
• Contract tests – using pact or custom scripts to verify:
  • Leaf value appears in telemetry under the expected metric name.
  • Alert rules fire correctly when the leaf crosses a threshold.
  • Downstream playbooks can read the leaf and act without error.
• Post‑commit validation – after applying a NETCONF <edit-config>:
  • <get> or <get-data> to retrieve the leaf and confirm its value.
  • gNMI Subscribe to stream the leaf briefly and verify no gaps.
  • Query the storage backend to confirm the field is present and correctly typed.
  • Dry‑run the alerting engine (e.g., amtool test alert for Prometheus) to ensure no false positives/negatives.

Each step returns a clear PASS/FAIL status and emits structured logs (JSON) for CI consumption.

Implementing YANG Leaf Introductions and Renamings

Using YANG Data Modeling for Leaf Introduction

To keep the change backward‑compatible:

• Place the leaf in a new container or list if it does not belong to an existing node.
• Assign a default value only when semantics allow; otherwise, set mandatory false and config true (or false for operational data).
• Choose a type matching the data semantics (e.g., uint32 for counters, string for descriptors, inet:ip-address for addresses). Avoid binary or anyxml unless essential.
• Add a description and reference stating the intended use (e.g., “Threshold for interface error rate used by the interface-error-high alert”).
• For operational‑only leaves, locate them under /oper-data and treat them as read‑only in normalization code.

Renaming Path Mappings with Minimal Disruption

Renaming is a breaking change for hard‑coded consumers. Limit impact by:

1. Introduce an alias – keep the old path as a leafref or choice pointing to the new leaf, marked deprecated true.
2. Version the external model – increment its version and document the mapping change in release notes.
3. Define a deprecation timeline – e.g., 90‑day sunset after which the alias is removed; note this in the YANG description and external docs.
4. Update normalization code – accept both old and new external field names, map them to the same internal YANG leaf, and log a warning when the old name is used.

Example YANG Module Code for Leaf Introduction

module acme-interface-telemetry {
  namespace "http://example.com/acme/interface/telemetry";
  prefix ait;

  import ietf-inet-types {
    prefix inet;
  }
  import ietf-yang-types {
    prefix yang;
  }

  organization "Acme Corp.";
  contact "netdev@example.com";
  description
    "Telemetry model for interface performance metrics.
     This version adds the 'error-rate-threshold' leaf for
     alerting purposes.";

  revision 2025-09-26 {
    description
      "Initial addition of error-rate-threshold leaf.";
    reference "ACME‑TEL‑2025‑001";
  }

  container interfaces {
    list interface {
      key "name";
      leaf name {
        type string;
      }
      leaf ifIndex {
        type uint32;
      }
      leaf admin-status {
        type enumeration {
          enum up { value 1; }
          enum down { value 2; }
        }
      }
      leaf oper-status {
        type enumeration {
          enum up { value 1; }
          enum down { value 2; }
          enum testing { value 3; }
        }
      }
      leaf in-errors {
        type yang:counter32;
        description "Number of inbound packets with errors.";
      }
      leaf out-errors {
        type yang:counter32;
        description "Number of outbound packets with errors.";
      }
      /* NEW LEAF */
      leaf error-rate-threshold {
        type uint32 {
          range "1..100";
        }
        units "percent";
        description
          "Configured error‑rate threshold (in percent) that
           triggers the interface-error-high alert.";
        default 5;
      }
    }
  }
}

Notes:

• The leaf is optional (mandatory false by default) and has a sensible default, preserving existing configurations.
• The description explicitly ties the leaf to an alert, aiding verification.

Example CLI Commands for Renaming Path Mappings

Assuming the external model previously used error_thr and we now prefer error-rate-threshold in YANG, we retain a deprecated leaf:

  leaf error-thr {
    type uint32 {
      range "1..100";
    }
    units "percent";
    description
      "DEPRECATED: Use error-rate-threshold instead.
       This leaf will be removed in release 2.0.";
    status deprecated;
  }

NETCONF workflow using ncclient (bash):

# 1. Validate the candidate module against the device's schema
ncclient --host router01 --username admin --password secret \
    --validate --candidate < acme-interface-telemetry.yang

# 2. Apply the configuration: add the new leaf while keeping the deprecated one
ncclient --host router01 --username admin --password secret \
    --edit-config --candidate <<EOF
<config xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <interfaces xmlns="http://example.com/acme/interface/telemetry">
    <interface>
      <name>GigabitEthernet0/0/0/0</name>
      <!-- Existing leaves omitted for brevity -->
      <error-rate-threshold>7</error-rate-threshold>
      <!-- Deprecated leaf can still be set; ignored by new code -->
      <error-thr>7</error-thr>
    </interface>
  </interfaces>
</config>
EOF

# 3. Verify the leaf is present and correctly valued
ncclient --host router01 --username admin --password secret \
    --get --filter "<interfaces xmlns='http://example.com/acme/interface/telemetry'>
                     <interface><name>GigabitEthernet0/0/0/0</name>
                      <error-rate-threshold/>
                     </interface>
                   </interfaces>"

These steps illustrate a verification‑gated approach: validate, commit in a controlled transaction, then run automated checks before promoting the change to a broader fleet. By following this workflow, you can introduce new YANG leaves, rename path mappings, or adjust normalization code without jeopardizing alerts, storage contracts, or downstream automation.