Canary shared templates without amplifying inherited mistakes

Introduction to Canary Strategy for Template Inheritance

Template inheritance lets a parent template define reusable configuration blocks while child templates or device‑specific variables fill in the gaps. In large estates a single parent may be inherited by hundreds or thousands of devices; a change to the parent fans out to every child, turning a subtle syntax error into a network‑wide outage.

A canary strategy isolates the first render to a small, representative subset of devices, verifies that the rendered children are safe, and only then allows the parent change to proceed to the broader group. The approach treats the template change as a transaction with explicit pre‑checks, a commit boundary, verification gates, and a defined rollback path.

Verification is the linchpin because:

• Render‑time errors are invisible until applied (e.g., malformed ACLs, duplicate IPs).
• Blast radius grows with inheritance depth.
• Operator confidence requires measurable proof before a fleet‑wide push.
• Rollback windows are often operational, not automatic, so verification must happen before commitment.

By anchoring the canary to explicit verification gates we convert a trust‑based push into a risk‑controlled sequence that can be halted at the first sign of trouble.

Designing a Canary Strategy

Identifying Key Template Components

Dissect the parent template to understand which sections drive risk:

Create a risk matrix that scores each component by likelihood of error and impact severity; focus canary verification on high‑risk items.

Creating a Canary Template

The canary template is a thin wrapper around the parent that forces a dedicated variable set and limits inheritance scope:

{# canary_parent.j2 #}
{% set _canary_mode = true %}
{% include "parent.j2" ignore missing %}

• _canary_mode – flag child templates can read to skip risky sections (e.g., production‑only BGP peers).
• ignore missing – prevents accidental inclusion of non‑existent snippets during early testing.
• Keep the canary template in a separate Git branch or protected folder; promote it to the parent branch only after verification.

Setting Up a Test Device Group

Select a canary device group that satisfies:

• Representativeness – mix of hardware models, OS versions, and feature sets reflecting the broader estate.
• Isolation – can be taken out of service or placed in a maintenance window without affecting production traffic.
• Size – small enough to limit blast radius (typically 1‑5 % of the total group, but never fewer than 3 devices to avoid statistical fluke).

Define the group in your inventory source (e.g., Ansible, Nornir, NetBox) with a label such as canary_<template_name>. Apply strict pre‑change checks:

1. Baseline capture – save running‑config and relevant operational state (e.g., show ip route summary, show interface status).
2. Pre‑change validation – run compliance scripts (Batfish, Cisco NSO NSO‑check, custom Python) to confirm the baseline meets design intent.
3. Change‑window approval – require a ticket or change‑control approval that explicitly lists the canary group as the target.

Only after these preconditions are satisfied do we proceed to the commit boundary – the point at which the rendered config is pushed to the canary devices.

Implementing the Canary Strategy

Using CLI to Apply Templates

Most network automation frameworks expose a CLI‑like interface for pushing config. Below is an example using Ansible with the ios_config module; the same principles apply to Nornir, SaltStack, or custom Python scripts.

# 1. Render the canary template to a temporary file
ansible-playbook -i inventory.yml render_canary.yml \
  -e "template=canary_parent.j2 output_dir=/tmp/canary"

# 2. Push the rendered config to the canary device group
ansible-playbook -i inventory.yml deploy_canary.yml \
  -e "canary_group=canary_bgp_policy config_dir=/tmp/canary"

render_canary.yml (simplified):

- name: Render canary template
  hosts: localhost
  gather_facts: false
  vars:
    template_name: "{{ template }}"
    output_dir: "{{ output_dir }}"
  tasks:
    - name: Load Jinja2 template
      template:
        src: "{{ template_name }}.j2"
        dest: "{{ output_dir }}/{{ inventory_hostname }}.cfg"
      delegate_to: localhost
      run_once: true

deploy_canary.yml:

- name: Deploy canary config
  hosts: "{{ canary_group }}"
  gather_facts: false
  tasks:
    - name: Load rendered config
      slurp:
        src: "/tmp/canary/{{ inventory_hostname }}.cfg"
      delegate_to: localhost
      register: cfg_file

    - name: Push config via CLI (dry‑run first)
      ios_config:
        src: "{{ cfg_file.content | b64decode }}"
        commit: false          # dry‑run; no write to startup-config
      register: dry_result

    - name: Abort if dry‑run fails
      fail:
        msg: "Dry‑run failed on {{ inventory_hostname }}: {{ dry_result.msg }}"
      when: dry_result.failed

    - name: Commit change (only after dry‑run succeeds)
      ios_config:
        src: "{{ cfg_file.content | b64decode }}"
        commit: true
      when: not dry_result.failed

Transaction boundary: The commit: false dry‑run is the pre‑commit verification gate. If any device fails the dry‑run, the playbook halts before any device is actually changed, preserving the original state.

Example Code for Template Rendering

If you prefer a pure Python rendering step (useful when integrating with a CI pipeline), Jinja2 can be invoked directly:

#!/usr/bin/env python3
import json
import jinja2
import sys
from pathlib import Path

def render_template(template_path: str, vars_path: str, output_path: str):
    env = jinja2.Environment(
        loader=jinja2.FileSystemLoader(searchpath=Path(template_path).parent),
        autoescape=jinja2.select_autoescape(['j2'])
    )
    template = env.get_template(Path(template_path).name)

    with open(vars_path) as f:
        vars_data = json.load(f)

    rendered = template.render(**vars_data)

    Path(output_path).write_text(rendered)
    print(f"Rendered config written to {output_path}")

if __name__ == "__main__":
    if len(sys.argv) != 4:
        print("Usage: render.py <template.j2> <vars.json> <output.cfg>")
        sys.exit(1)
    render_template(sys.argv[1], sys.argv[2], sys.argv[3])

Usage in a canary pipeline:

render.py canary_parent.j2 canary_vars.json /tmp/canary/r1.cfg

Verifying Rendered Template Children

Verification must happen after the dry‑run but before the commit. Typical checks include:

If any check fails, the playbook triggers the rollback trigger (see next section) and alerts the operator via the change‑management ticketing system.

Troubleshooting Common Issues

Debugging Template Rendering Errors

1. Undefined variable – Jinja2 raises UndefinedError. Enable strict=True in the environment to fail fast.

   env = jinja2.Environment(loader=..., undefined=jinja2.StrictUndefined)

2. Whitespace surprises – Use {% set __ = "" %} or control newline injection with jinja2.select_autoescape. Render to a temporary file and inspect with cat -A.
3. Incorrect filter output – Add a debug step that prints the intermediate value:

   {{ my_var | ipaddr('net') | debug }}

   (Create a custom debug filter that returns the value and logs it.)

Identifying and Resolving Device Group Conflicts

• Overlapping groups – A device may belong to both the canary group and a production group that receives a different parent version. Use inventory queries to assert group_intersection == empty.
• Variable precedence collisions – If the same variable is defined in group_vars, host_vars, and extra_vars, the most specific wins. Document the precedence order and enforce it via a linting step (e.g., ansible-lint with var_precedence rule).

Handling Template Inheritance Conflicts

When a child template overrides a parent block, the effective result can be non‑obvious. Strategies:

• Explicit block naming – Name every overridable block in the parent ({% block interface_config %}) and require children to use the same name.
• Render diff report – After rendering, compute a diff between the parent‑only output and the child‑overridden output. Highlight any overridden blocks for manual review.
• Version lock – Store a hash of the parent template alongside each child’s rendered output. If the parent hash changes, flag the child for re‑validation.

Scaling the Canary Strategy

Limitations of Template Inheritance

• Depth explosion – Deep inheritance chains (parent → intermediate → child) increase the surface area for hidden bugs.
• Variable shadowing – Variables defined at multiple layers can silently override each other, making impact analysis hard.
• Render time – Large templates with heavy loops can take seconds to render per device, slowing the canary cycle.

Scaling Template Rendering for Large Device Groups

• Batch rendering – Render configs for a batch of devices (e.g., 50) in a single Jinja2 call using a list context, then split the output. This reduces interpreter overhead.
• Parallel execution – Use a process pool (concurrent.futures.ProcessPoolExecutor) to render multiple host contexts simultaneously.
• Incremental rendering – If only a subset of variables changed, cache the rendered template skeleton and only re‑apply the variable substitution step.

Optimizing Template Verification for Performance

• Pre‑compute static checks – Run syntax and lint checks on the template itself once per commit; reuse the result across all devices.
• Selective validation – Apply heavyweight verification (Batfish, OPA) only to devices that represent distinct feature sets (e.g., one per line card type).
• Result caching – Store verification outcomes keyed by a hash of the rendered config; if two devices receive identical rendered output, reuse the prior verification result.

Code Examples and CLI Commands

Rendering Templates using CLI

# Using ansible-playbook with the template module (render only)
ansible localhost -m template -a "src=canary_parent.j2 dest=/tmp/canary/{{ inventory_hostname }}.cfg"

Verifying Template Children using API Calls

Assuming a REST‑based config validation service (e.g., Batfish service):

#!/usr/bin/env bash
DEVICE=$1
CONFIG_FILE=$2
VALIDATION_URL="https://batfish.example.com/api/validate"

response=$(curl -s -X POST "$VALIDATION_URL" \
     -H "Content-Type: application/json" \
     -d @<(jq -n \
          --arg hostname "$DEVICE" \
          --arg config "$(cat $CONFIG_FILE)" \
          '{hostname: $hostname, config: $config}'))

echo "$response" | jq .
# Expect {"status":"PASS", "details":{...}}

Example Scripts for Automating Canary Strategy

A concise Bash orchestrator that ties rendering, dry‑run, verification, and commit together:

#!/usr/bin/env bash
set -euo pipefail

# ----- INPUTS -----
TEMPLATE="canary_parent.j2"
VARS_FILE="canary_vars.json"
GROUP="canary_bgp_policy"
WORKDIR="/tmp/canary_$(date +%s)"
mkdir -p "$WORKDIR"

# ----- 1. RENDER -----
echo "[*] Rendering templates..."
ansible localhost -m template -a "src=$TEMPLATE dest=$WORKDIR/{{ inventory_hostname }}.cfg" \
    -e "vars_file=$VARS_FILE"

# ----- 2. DRY‑RUN & VERIFY -----
echo "[*] Performing dry‑run and verification..."
ansible -i inventory.ini "$GROUP" -m ios_config -a "
    src=$WORKDIR/{{ inventory_hostname }}.cfg
    commit=false
" | tee dryrun.log

if grep -q "failed" dryrun.log; then
    echo "[!] Dry‑run failed – aborting"
    exit 1
fi

# ----- 3. VERIFICATION (example: Batfish) -----
echo "[*] Running semantic verification..."
# placeholder for verification step; replace with actual tool
# e.g., batfish_verify.py $WORKDIR/*.cfg

# ----- 4. COMMIT -----
echo "[*] Committing changes..."
ansible -i inventory.ini "$GROUP" -m ios_config -a "
    src=$WORKDIR/{{ inventory_hostname }}.cfg
    commit=true
"
echo "[*] Canary rollout complete"