LinkState
Go back

Bad config or bad gate or bad sample

Introduction to Canary Releases

Definition and Purpose

A canary release is a deployment strategy in which a new version of software (or configuration) is rolled out to a small subset of production traffic—often called the canary group—before being promoted to the full fleet. The purpose is to detect regressions, performance degradations, or configuration errors early, while limiting the blast radius of any faulty change.

Benefits and Challenges

Benefits

Understanding Canary Trips

Definition and Implications

A canary trip occurs when the canary group fails one or more verification gates, triggering an automatic halt or alert that prevents further promotion. The trip signals that something in the canary deployment is inconsistent with the expected behavior defined by the verification criteria. Implications

Common Causes of Canary Trips

  1. Broken rendered configuration – the configuration that actually reaches the device or process is malformed, missing, or contains syntax errors.
  2. Broken verification logic – the health‑check scripts, metric thresholds, or test suites used to judge the canary are incorrectly implemented or rely on stale assumptions.
  3. Unrepresentative traffic sampling – the canary receives traffic that does not mirror the production mix (e.g., only internal users, or a skewed geographic distribution), causing metrics to deviate from expected baselines.

Plausible Causes of Canary Trips

Broken Rendered Config

Definition and Examples

The rendered config is the final artifact produced by a templating or generation engine (e.g., Jinja2, Helm, Ansible templates) that is pushed to the target system. A break can happen when:

neighbor {{ peer_ip }} remote-as {{ local_as }}

which is invalid syntax for the router and causes the BGP session to fail, tripping the canary.

Troubleshooting Steps

  1. Retrieve the rendered config from the canary node (e.g., via show running-config or a file dump).
  2. Diff it against the last known good rendered config (or the template with known good variable values).
  3. Validate syntax using the native parser (cisco_ios check config, Juniper commit check, frr vtysh -c "show running-config").
  4. Inspect variable sources (inventory, CI pipeline, feature flags) for missing or mistyped values.
  5. Re‑render locally with the exact same variable set and compare output.

Code Examples for Verification

Python snippet to render a Jinja2 template and validate with a simple regex

import jinja2
import subprocess
import sys

template_str = """ 
neighbor {{ peer_ip }} remote-as {{ local_as }} 
"""

def render_and_check(vars_dict):
    tmpl = jinja2.Template(template_str)
    rendered = tmpl.render(**vars_dict)
    print("Rendered config:\n", rendered)
    # Basic sanity check: neighbor line must contain an IP address
    import re
    if not re.search(r'neighbor\s+\d{1,3}(\.\d{1,3}){3}\s+remote-as\s+\d+', rendered):
        print("ERROR: Rendered config failed sanity check")
        return False
    return True

if __name__ == "__main__":
    vars = {"local_as": 65001}  # Intentionally missing peer_ip to demonstrate failure
    if not render_and_check(vars):
        sys.exit(1)

CLI verification on a Cisco IOS‑XR device

# Pull the running config from the canary router
show running-config | section neighbor
# Validate syntax (no commit, just parse)
show configuration failed

If the parser reports an error at the line containing {{ peer_ip }}, the cause is a broken rendered config.

Broken Verification Logic

Definition and Examples

Verification logic comprises the scripts, metric queries, or test suites that decide whether the canary is healthy. A break can be:

Troubleshooting Steps

  1. Extract the verification command/script from the pipeline definition (e.g., Jenkinsfile, GitLab CI YAML).
  2. Execute it manually on a canary host (or a replica) with the same environment variables.
  3. Capture stdout/stderr and exit code; compare against expected success criteria.
  4. Instrument the script with set -x (bash) or print statements to see which branch is taken.
  5. Check metric sources directly (e.g., curl http://prometheus:9090/api/v1/query?query=up{job="canary"}) to ensure the query returns data.
  6. Review recent changes to the verification repository (git blame) to spot the offending commit.

CLI Examples for Debugging

Bash health‑check with debugging

#!/usr/bin/env bash
set -euo pipefail
HEALTH_URL="http://localhost:8080/health"
EXPECTED='"status":"ok"'
echo "[$(date)] Checking $HEALTH_URL"
response=$(curl -sSf "$HEALTH_URL") || { 
  echo "ERROR: curl failed"
  exit 1
}
echo "Response: $response"
if [[ "$response" =~ $EXPECTED ]]; then
  echo "HEALTH CHECK PASSED"
  exit 0
else
  echo "HEALTH CHECK FAILED: expected '$EXPECTED' not found"
  exit 1
fi

Run it with bash -x health_check.sh to see each step. Prometheus query sanity check

# Replace with actual job name used in canary
curl -g "http://prometheus:9090/api/v1/query?query=up{job=\"canary_service\"}"

If the result is "result":[], the query is broken (label mismatch or metric not exposed).

Unrepresentative Traffic Sampling

Definition and Examples

The canary receives a subset of live traffic intended to mirror production. If the sampling mechanism is biased, the observed metrics will not reflect the true behavior of the new version under real load, leading to either false trips or missed defects. Common sources of bias:

Troubleshooting Steps

  1. Identify the traffic‑splitting mechanism (e.g., LB weighting, service mesh traffic split, ingress header).
  2. Collect samples of incoming requests on the canary (e.g., via tcpdump, ngrep, or access logs) and compare attributes (source IP, user‑agent, geolocation, headers) against a baseline from the stable fleet.
  3. Measure request rate and distribution (e.g., using tcptrace or tshark -z io,stat,1,"COUNT(http.request.method) ip.src").
  4. Validate the split percentage by counting packets/flows directed to canary vs. stable over a fixed interval.
  5. If using a service mesh (Istio/Linkerd), inspect the virtual service or traffic split policy and check the actual weights via the mesh’s telemetry (e.g., istioctl proxy-config routes <pod>).

Code Snippets for Traffic Analysis

Python with scapy to sample and classify traffic

from scapy.all import sniff, IP, TCP
from collections import Counter

canary_ip = "10.0.5.20"
stable_ip = "10.0.5.30"

def pkt_handler(pkt):
    if IP in pkt and TCP in pkt:
        src = pkt[IP].src
        dst = pkt[IP].dst
        if dst == canary_ip:
            counter["canary"] += 1
        elif dst == stable_ip:
            counter["stable"] += 1

counter = Counter()
sniff(filter="tcp port 80 or tcp port 443", prn=pkt_handler, timeout=30)
print("Canary packets:", counter["canary"])
print("Stable packets:", counter["stable"])
print("Canary share:", counter["canary"] / (counter["canary"] + counter["stable"]))

If the canary share deviates far from the configured weight (e.g., 5 % vs. expected 20 %), the sampling is unrepresentative. NGINX ingress log analysis

# Assume logs are combined format with $canary flag
awk '$9 ~ /canary/ {print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -nr

Compare the distribution of $http_user_agent or $geoip_country_code between canary and stable logs.

Designing Tests to Eliminate Causes

Test Design Principles

Test Cases for Broken Rendered Config

Automated Testing Examples

GitLab CI job that renders templates and runs a syntax check

render_config:
  stage: test
  image: python:3.11-slim
  script:
    - pip install jinja2
    - python - <<'PY'
      import jinja2, sys, os, subprocess
      tmpl_path = "templates/bgp.j2"
      with open(tmpl_path) as f:
        tmpl = jinja2.Template(f.read())
      # Load vars from the same source used in production (e.g., vars.yml)
      import yaml
      with open("vars/production.yml") as f:
        vars = yaml.safe_load(f)
      rendered = tmpl.render(**vars)
      out_path = "rendered/bgp.conf"
      with open(out_path, "w") as f:
        f.write(rendered)
      # Run vendor‑specific syntax checker (example: FRR)
      result = subprocess.run(["vtysh", "-c", f"show running-config"], capture_output=True, text=True)
      if result.returncode != 0:
        print("Syntax check failed:", result.stderr)
        sys.exit(1)
      print("Rendered config syntax OK")
  PY
  artifacts:
    paths:
      - rendered/

If the template contains an undefined variable, the Jinja2 render will raise an exception and the job fails, preventing the broken config from ever reaching the canary.

Manual Testing Scenarios

  1. Variable‑absence test – Temporarily delete a variable from the vars file, render, and verify that the resulting config fails the device’s parser.
  2. Conditional‑block test – Flip a boolean flag that controls inclusion of a critical section (e.g., ACL) and confirm the rendered output either includes or excludes it as expected.
  3. Encoding test – Introduce a Windows‑style line ending (\r\n) into a template snippet and verify that the final file still parses (or fails, depending on device tolerance).

Test Cases for Broken Verification Logic

Unit Testing Examples

Python unittest for a health‑check function

import unittest
from unittest.mock import patch, MagicMock
import healthcheck  # module containing check_service()

class TestHealthCheck(unittest.TestCase):
    @patch("healthcheck.requests.get")
    def test_success(self, mock_get):
        mock_resp = MagicMock()
        mock_resp.text = '{"status":"ok"}'
        mock_resp.status_code = 200
        mock_get.return_value = mock_resp
        self.assertTrue(healthcheck.check_service())

    @patch("healthcheck.requests.get")
    def test_failure_wrong_status(self, mock_get):
        mock_resp = MagicMock()
        mock_resp.text = '{"status":"error"}'
        mock_resp.status_code = 200
        mock_get.return_value = mock_resp
        self.assertFalse(healthcheck.check_service())

    @patch("healthcheck.requests.get")
    def test_network_error(self, mock_get):
        mock_get.side_effect = ConnectionError("DNS failure")
        self.assertFalse(healthcheck.check_service())

if __name__ == "__main__":
    unittest.main()

Run this test in the CI pipeline; any change that breaks the function (e.g., hard‑coding a wrong URL) will cause the test to fail, catching the problem before the canary sees it.

Integration Testing Scenarios

  1. End‑to‑end verification in a staging cluster – Deploy the canary version to a isolated namespace, drive synthetic traffic via a tool like hey or wrk, and run the exact verification script used in production. Assert that the script exits with code 0 under known‑good conditions.
  2. Fault injection – Temporarily break the service (e.g., stop the backend pod) and confirm that the verification script correctly returns a non‑zero exit code.
  3. Metric‑query validation – Deploy a temporary Prometheus instance, push known metric values via the Pushgateway, then run the verification’s PromQL query and check that the evaluation matches the expected threshold.

Test Cases for Unrepresentative Traffic Sampling

Load Testing Examples

Using Locust to shape traffic and verify split

from locust import HttpUser, task, between

class BrowserUser(HttpUser):
    wait_time = between(1, 3)

    @task
    def home(self):
        self.client.get("/")

Run Locust with a header that marks canary traffic:

locust -f locustfile.py --headless -u 200 -r 20 --run-time 5m \
  --host https://example.com \
  --header "X-Canary: true"

On the ingress controller, enable logging of the X-Canary header and verify the traffic distribution matches the expected split percentage.

Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email
Previous Post
Symbolic analysis versus packet probes for policy CI
Next Post
Default-Gateway Community Is Not Reachability