Partial rollback created a half-upgraded fabric

Introduction to Blameless Reviews

Definition and Purpose

A blameless review is a structured, evidence‑based examination of an operational event that focuses on what happened, why it happened, and how the system can be made more resilient—without assigning fault to individuals. In network change automation, the review captures the full lifecycle of a change: intent, design, pre‑checks, execution, verification, rollback, and post‑change validation. Its purpose is to surface latent weaknesses in tooling, processes, and safeguards so that future changes are safer and more predictable.

Importance in Post‑Rollback Analysis

When a rollback succeeds on some devices but stalls on others, the network ends up in a split operational state—neither the old configuration nor the new one is fully understood or enforced. This condition can produce subtle forwarding anomalies that are difficult to trace, mask underlying bugs, and erode confidence in automation. A blameless review of such an incident provides:

• A factual timeline distinguishing root cause (why the rollback stalled) from amplifiers (e.g., monitoring gaps) and recovery mistakes (e.g., prematurely declaring success).
• Concrete data on transaction boundaries, verification gates, and blast‑radius controls that either held or failed.
• Actionable insights for improving idempotency, rollback mechanisms, and operator intervention paths without resorting to punitive measures.

Understanding the Rollback Scenario

Successful Rollback on Some Devices

A scheduled ACL refresh was pushed to 240 edge routers via an Ansible playbook using NETCONF <edit-config> with a confirmed commit (timeout 120 s). After validating the new ACL, a trigger initiated a rollback to the previous ACL version. On 162 devices the rollback completed within the confirmation window, the devices reverted to the pre‑change configuration, and post‑rollback verification (ACL hit‑count checks and traffic‑flow validation) passed.

Key observations

• Pre‑checks (configuration syntax validation, NETCONF capability exchange) succeeded.
• The commit boundary was honored: the confirmed commit automatically rolled back when no commit followed within the timeout.
• Verification gate (a script that queried show access-lists and compared SHA‑256 hashes) reported equality with the baseline.
• Blast radius was limited to the individual device; no traffic disruption was reported.

Stalled Rollback on Other Devices

On the remaining 78 devices the rollback did not finish. Symptoms included:

• NETCONF session remained in the confirmed‑commit state past the timeout, requiring manual intervention (<discard-changes> or session reset).
• CLI output showed the candidate datastore still holding the new ACL while the running datastore retained the old ACL, leading to a mixed state where some ACL entries were active and others were not.
• Monitoring polls (SNMP ifInErrors, ifOutErrors) began to rise after ~5 min, indicating packet drops caused by malformed ACL entries.
• Manual login and execution of rollback or reload restored a clean state, but the delay exceeded the planned rollback window (10 min).

Root‑cause hints from device logs

• Resource exhaustion – the device’s NETCONF agent hit an internal queue limit when processing a large ACL (≈ 4 k lines) during the confirmed‑commit phase.
• Version‑specific bug – a known defect in OS 7.3.2 where the confirmed‑commit timer is ignored if the configuration contains nested ip access-list extended statements with remark lines longer than 255 chars.
• Concurrent change – a separate, low‑priority SNMP‑set operation attempted to modify an interface description on the same device, causing a lock conflict on the candidate datastore.

Split Operational State and Its Implications

Because safeguards (e.g., route‑health checks, BGP session monitoring) only looked at aggregate metrics (overall loss < 0.1 %), they failed to flag the per‑device ACL corruption. The split state meant that old safeguards (baseline ACL assumptions) were violated on a subset of devices, while new safeguards (designed for the post‑change ACL) were not fully active, leaving a gap in both detection and mitigation.

Identifying Key Factors for Blameless Review

Technical Factors

Device and Network Configuration

• ACL size and complexity – the new ACL exceeded the recommended 2 k‑line threshold for the affected platform, increasing processing time during commit.
• Configuration nesting – heavy use of remark fields and nested ip access-list statements triggered the OS bug.
• Concurrent management traffic – SNMP‑set and Syslog‑push operations were allowed during the change window, creating lock contention.

Rollback Mechanism and Tools

• Confirmed‑commit reliance – the rollback assumed the timer would fire reliably; the bug prevented timer expiration.
• Lack of idempotent rollback – the playbook re‑applied the baseline ACL via <edit-config> rather than issuing an explicit rollback command, which on the affected OS required a separate <discard-changes> after timeout.
• No pre‑rollback health check – the playbook did not verify that the device was in a clean candidate state before initiating the confirmed commit.

Safeguards and Monitoring Systems

• Metric granularity – loss‑rate thresholds were calculated at the aggregation layer; per‑device ACL error counters were not correlated with traffic impact.
• Alert suppression – syslog severity for ACL lookup failures was set to informational, preventing paging.
• Verification gate limitation – the post‑rollback script only checked the running datastore; it did not inspect the candidate datastore for stray changes.

Organizational Factors

Communication and Coordination

• Change window overlap – the ACL refresh was scheduled concurrently with a routine SNMP‑poll‑based firmware inventory, which was not communicated to the change‑owner.
• Shift handoff – the engineer who initiated the rollback left their shift before the stalled devices were identified; the incoming team lacked visibility into the ongoing NETCONF sessions.

Change Management Processes

• Pre‑change risk assessment – the change request listed the ACL size as “medium” and did not trigger a deeper review for devices running OS 7.3.2.
• Rollback test coverage – lab validation used a small ACL (≤ 500 lines) on a single device model, missing the scaling and bug interaction.

Training and Expertise

• NETCONF troubleshooting – junior engineers were unfamiliar with interpreting <rpc-error> tags related to queue overruns.
• Device‑specific quirks – knowledge of the OS 7.3.2 confirmed‑commit bug resided only in a senior engineer’s personal notes, not in the team runbook.

Conducting the Blameless Review

Gathering Information and Data

Log Analysis and Error Messages

• Collected NETCONF session logs (/var/log/netconfd) from all devices; filtered for rpc-error with error-type protocol and error-tag operation-failed.
• Extracted syslog entries matching %ACL-4-ACL_LOOKUP_FAIL and %SYS-5-CONFIG_I during the 15‑minute window.
• Retrieved audit trails from the Ansible controller (/var/log/ansible/ansible.log) to confirm which hosts received the confirmed‑commit RPC.

Network and Device Performance Metrics

• Pulled SNMP counters (ifInErrors, ifOutErrors, ipInDiscards) via Prometheus every 30 s; computed per‑device error rates.
• Queried NetFlow/IPFIX for ACL‑related drop flows (acl-id field) to see if drops correlated with specific ACL lines.
• Reviewed device CPU/memory (show processes cpu, show memory statistics) to confirm resource exhaustion on stalled units.

User and Administrator Feedback

• Surveyed help‑desk tickets for the affected time slice; noted intermittent “cannot reach internal portal” reports matching the sub‑net served by the stalled devices.
• Conducted a short debrief with the on‑call network engineer to capture their perception of the rollback progress and any manual actions taken.

Analyzing the Data and Identifying Patterns

Correlating Success and Failure Factors

• Success correlation – devices with ACL ≤ 1500 lines, OS ≥ 7.5.0, and no concurrent SNMP‑set completed rollback within 45 s.
• Failure correlation – all stalled devices ran OS 7.3.2, had ACL > 2500 lines with long remarks, and experienced an SNMP‑set lock within ± 10 s of the confirmed‑commit start.

Identifying Bottlenecks and Single Points of Failure

• The NETCONF agent’s internal queue emerged as a bottleneck; logs showed QUEUE_FULL errors precisely when the confirmed‑commit timer should have fired.
• The configuration lock (candidate datastore) was a single point of failure: any concurrent write blocked the commit/rollback sequence, causing the timer to be ignored.

Assessing Safeguard Effectiveness

• Verification gate (running‑datastore hash) passed on stalled devices because the hash excluded the candidate datastore; thus the gate gave a false sense of completeness.
• Monitoring lacked a device‑level ACL integrity check that would have flagged the candidate‑running mismatch (e.g., show configuration commit pending).
• Alerting thresholds were too high; the error rate increase on individual devices remained below the aggregate trigger.

Troubleshooting the Rollback Issues

Debugging Techniques and Tools

Using CLI Commands for Troubleshooting

# Verify the state of the candidate vs running datastore
show configuration commit pending          # lists uncommitted changes
show configuration | match access-list   # current running ACL
show configuration datastore candidate | match access-list  # candidate ACL

# Check NETCONF session status
show netconf sessions detail | include <session-id>

# Force a discard of the candidate datastore (if supported)
netconf console --host <device> --username admin --password <pwd> \
    --rpc '<discard-changes/>'

# Validate ACL syntax after manual fix
show ip access-list <name> | include remark

These commands allow an operator to confirm whether a device is stuck in a confirmed‑commit state, view the divergent configurations, and manually issue a discard or commit to resolve the stalemate.

Analyzing Network Traffic and Device Logs

• Capture packet traces on affected interfaces during the error window (tcpdump -i eth0 -w acl_drop.pcap) to observe packets being dropped due to malformed ACL entries.
• Parse syslog with a simple script to count %ACL-4-ACL_LOOKUP_FAIL per device and correlate with timestamps of the confirmed‑commit RPC.

Code Examples for Automated Rollback and Monitoring

Scripting Rollback Processes (Ansible + NETCONF)

---
- name: Rollback ACL with safety checks
  hosts: edge_routers
  gather_facts: false
  vars:
    acl_baseline: "{{ lookup('file', 'acls/baseline.cfg') }}"
    confirm_timeout: 120
  tasks:
    - name: Ensure no pending commit
      iosxr_netconf:
        host: "{{ inventory_hostname }}"
        username: "{{ netconf_user }}"
        password: "{{ netconf_pass }}"
        rpc: "<get><configuration><commit><pending/></commit></configuration></get>"
      register: pending
      failed_when: pending.xml is search('<pending>')   # abort if pending changes exist

    - name: Load baseline configuration (candidate)
      iosxr_netconf:
        host: "{{ inventory_hostname }}"
        username: "{{ netconf_user }}"
        password: "{{ netconf_pass }}"
        config: "{{ acl_baseline }}"
        format: text
        operation: replace   # replace candidate datastore

    - name: Issue confirmed commit
      iosxr_netconf:
        host: "{{ inventory_hostname }}"
        username: "{{ netconf_user }}"
        password: "{{ netconf_pass }}"
        rpc: "<commit><confirmed><timeout>{{ confirm_timeout }}</timeout></confirmed></commit>"
      register: commit_result
      # Additional verification tasks would follow here

This playbook adds a pre‑check for pending changes, loads the baseline into the candidate datastore, and issues a confirmed commit with a configurable timeout.

Enhanced Verification Gate (Python‑like pseudocode)

def verify_acl_consistency(device):
    running = device.running_config.get('access-list')
    candidate = device.candidate_config.get('access-list')
    if running != candidate:
        raise Alarm("ACL mismatch between running and candidate datastores on %s" % device.hostname)
    # Optional: compute hash and compare to baseline
    if hash(running) != BASELINE_HASH:
        raise Alarm("Running ACL does not match baseline on %s" % device.hostname)

This gate checks both datastores, preventing the false‑positive scenario observed in the incident.

End of review.