Introduction to Maintenance Window Validation

Operational question: Did the maintenance activity declared in a change ticket actually remain inside the approved time window, or did it drift earlier/later and risk impacting production traffic?

A data‑first view starts with a timeline panel that overlays three signals:

Plotting these boolean series side‑by‑side in Grafana instantly reveals any overlap or gap. If drain_active starts before the change_ticket_window or ends after it, a boundary violation is flagged.

What the panel proves: temporal coincidence of the three signals.
What it cannot prove: whether the drain was effective (traffic actually stopped) or whether the change ticket accurately reflects the intended scope (which interfaces were to be drained). Those gaps require additional signals: interface counters, flow logs, and control‑plane state.

Overview of Key Components

• Intended state – the change ticket (what was approved).
• Rendered state – the scheduled drain event (what the automation was told to do).
• Applied state – the actual drain execution observed in logs/traces (what happened on the wire).
• Observed state – the metrics/alerts that surface the symptom to operators (e.g., latency spikes, packet loss).

Validation succeeds when Intended → Rendered → Applied → Observed form a contiguous, time‑aligned chain with no gaps or overlaps outside the declared window.

Understanding Change Tickets

Definition and Purpose

A change ticket is the source‑of‑truth record of an approved modification. In ITSM tools (ServiceNow, Jira Service Management, Remedy) it captures:

• Unique identifier (ticket_id)
• Planned start/end timestamps (window_start, window_end) in ISO‑8601 UTC
• Affected CI list (routers, switches, servers)
• Approval chain and risk assessment
• Rollback procedure

Structure and Content

Typical JSON payload exported via ITSM REST API:

{
  "ticket_id": "CHG-2024-09876",
  "state": "approved",
  "window_start": "2024-09-20T02:00:00Z",
  "window_end":   "2024-09-20T04:30:00Z",
  "affected_cis": [
    {"ci_id": "rtr01", "type": "router", "interfaces": ["Eth1/1","Eth1/2"]},
    {"ci_id": "rtr02", "type": "router", "interfaces": ["Eth1/1"]}
  ],
  "description": "Scheduled BGP peer drain for link upgrade",
  "approvals": ["net-eng lead", "change advisory board"]
}

Example: Retrieving a Ticket

curl -s -H "Authorization: Bearer $TOKEN" \
     https://itsm.example.com/api/now/table/change_request?sys_id=CHG-2024-09876 \
   | jq '.result | {ticket_id, state, window_start, window_end, affected_cis}'

Output (pretty‑printed):

{
  "ticket_id": "CHG-2024-09876",
  "state": "approved",
  "window_start": "2024-09-20T02:00:00Z",
  "window_end": "2024-09-20T04:30:00Z",
  "affected_cis": [
    {"ci_id":"rtr01","type":"router","interfaces":["Eth1/1","Eth1/2"]},
    {"ci_id":"rtr02","type":"router","interfaces":["Eth1/1"]}
  ]
}

What the ticket proves: the intended window and scope.
What it cannot prove: whether the automation actually scheduled a drain for exactly those interfaces at exactly those times. That requires inspecting the rendered state (scheduled drain event).

Scheduled Drain Events

Definition and Purpose

A scheduled drain event is the rendered state: the concrete automation job that tells network devices to gracefully steer traffic away from a set of interfaces (or entire nodes) for a defined interval. It is usually driven by an orchestrator (Ansible, Terraform, custom Python) that consumes the change ticket and issues gNMI/NETCONF set operations or CLI scripts.

Types of Drain Events

Configuring and Triggering Drain Events (Ansible Example)

- name: Load change ticket
  uri:
    url: "https://itsm.example.com/api/now/table/change_request/{{ ticket_id }}"
    method: GET
    headers:
      Authorization: "Bearer {{ itsm_token }}"
  register: ticket_resp

- name: Extract window and interfaces
  set_fact:
    window_start: "{{ ticket_resp.json.result.window_start }}"
    window_end:   "{{ ticket_resp.json.result.window_end }}"
    ifaces: "{{ ticket_resp.json.result.affected_cis | map(attribute='ci_id') | list }}"

- name: Build gNMI Set payload
  copy:
    dest: "/tmp/drain_{{ ticket_id }}.json"
    content: |
      {
        "update": [
          {
            "path": "interfaces/interface[name={{ item }}]/state/admin-status",
            "val": {"enum_val": 2}   // DOWN in OpenConfig
          }
        ]
      }
  loop: "{{ ifaces }}"

- name: Schedule drain via cron (at window_start)
  cron:
    name: "drain-{{ ticket_id }}"
    minute: "{{ window_start | date('%M') }}"
    hour:   "{{ window_start | date('%H') }}"
    day:    "{{ window_start | date('%d') }}"
    month:  "{{ window_start | date('%m') }}"
    weekday: "{{ window_start | date('%w') }}"
    job: "/usr/bin/gnmic -a {{ routers }} -u {{ user }} -p {{ pw }} set --file /tmp/drain_{{ ticket_id }}.json"

Example CLI Command for Scheduling a Drain Event

# Epoch seconds for window start and end
START=$(date -d "2024-09-20 02:00:00 UTC" +%s)
END=$(date -d "2024-09-20 04:30:00 UTC" +%s)

# Create an at‑job that runs the drain at START
echo "gnmic -a rtr01,rtr02 -u admin -p secret set \
  --update 'interfaces/interface[name=Eth1/1]/state/admin-status=DOWN' \
  --update 'interfaces/interface[name=Eth1/2]/state/admin-status=DOWN'" | at $START

# Create a second at‑job to restore interfaces at END
echo "gnmic -a rtr01,rtr02 -u admin -p secret set \
  --update 'interfaces/interface[name=Eth1/1]/state/admin-status=UP' \
  --update 'interfaces/interface[name=Eth1/2]/state/admin-status=UP'" | at $END

What the scheduled drain proves: the rendered state (what automation was told to do).
What it cannot prove: whether the device actually applied the configuration, or whether any transient flaps occurred before/after the scheduled times. Those answers live in the applied state (logs/traces).

Actual Trace and Log Timelines

Log Collection and Analysis Tools

• Syslog / JTI streamed to a centralized collector (e.g., Elasticsearch, Loki, Splunk).
• gNMI Subscribe for real‑time state (interfaces/interface/state/admin-status, interfaces/interface/state/counters).
• Telemetry pipeline (Telegraf → Prometheus, or OpenTelemetry Collector → Tempo/Grafana).

Trace Collection and Analysis Methods

• OpenTelemetry instrumentation on routing daemons (e.g., FRR, Juniper Junos telemetry agent) to generate spans for BGP peer state changes, interface admin changes.
• Correlation IDs injected by the orchestrator (e.g., x-change-id: CHG-2024-09876) and propagated via gNMI metadata.
• Trace backends (Jaeger, Tempo) enable querying spans by change_id to see the exact sequence of admin‑state changes.

Example Log and Trace Output

Syslog snippet (Loki query) for router rtr01:

2024-09-20T01:59:58.123Z rtr01 daemon.notice netconfd[1234]: <rpc-reply><msg>Configuration changed: set interfaces interface Eth1/1 admin-status down</msg></rpc-reply>
2024-09-20T02:00:01.456Z rtr01 daemon.info bgpd[5678]: %BGP-5-ADJCHANGE: neighbor 10.0.0.2 Down Interface flap
2024-09-20T04:29:59.789Z rtr01 daemon.notice netconfd[1234]: <rpc-reply><msg>Configuration changed: set interfaces interface Eth1/1 admin-status up</msg></rpc-reply>
2024-09-20T04:30:02.010Z rtr01 daemon.info bgpd[5678]: %BGP-5-ADJCHANGE: neighbor 10.0.0.2 Up Interface restored

Corresponding Tempo trace (filtered by change_id=CHG-2024-09876):

What the logs/traces prove: the applied state (what actually happened on the devices) and its timing relative to the intended and rendered states.

By correlating the three layers—ticket, scheduled automation, and observed telemetry—you can definitively answer whether a maintenance window stayed inside its declared boundary.