Commit confirm is not blast-radius control

Introduction to Commit‑Confirm and Risky Automation

Definition of Commit‑Confirm

Commit‑confirm is a two‑stage configuration operation supported by many network operating systems (NOS).

1. Commit – the candidate configuration is installed in the running datastore but marked unconfirmed.
2. Confirm – if an explicit confirm is received before a timer expires, the change becomes permanent; otherwise the NOS automatically rolls back to the pre‑commit state.

The mechanism protects against lock‑out scenarios where a mis‑configured remote session loses connectivity after a push.

Benefits and Limitations

Myth to test: “If we enable commit‑confirm on every automated push, the change is safe regardless of blast radius or verification.”
We will examine why this belief fails when automation touches multiple devices, when timers expire incorrectly, or when the underlying change is logically flawed.

Understanding Transaction Timers

Overview

A transaction timer (commit timer or confirm timeout) starts when a commit confirm operation is issued. The NOS installs the candidate as a pending configuration and begins a monotonic countdown. If the timer reaches zero without a confirm, the NOS automatically rolls back to the last known good configuration.

How Transaction Timers Work

1. Operator or automation issues commit confirm with timeout T.
2. NOS copies the candidate to a pending datastore and starts the timer.
3. While the timer runs:
   • The pending configuration is active on the forwarding plane (traffic uses the new settings).
   • The operator can issue commit to make the change permanent, or wait for timeout.
4. On timeout: NOS issues a rollback to the previous committed configuration, clears the pending datastore, and returns to the pre‑change state.
5. On explicit confirm before timeout: NOS marks the pending configuration as committed and cancels the timer.

Limitations in Risky Automation

• No semantic validation – timer measures elapsed time only; it does not verify that the new configuration satisfies service intent (correct ACLs, MTU, routing policy, etc.).
• Blind to cascading failures – a change on device A that breaks a BGP session with device B may cause A to roll back on timer expiry, while B may have already withdrawn routes, leading to transient black‑holes that persist until B’s own timers expire.
• Timer drift and NTP issues – clock skew across a large fleet can cause devices to roll back at different times, creating temporary inconsistency.
• Operator dependence in fully automated pipelines – automation must synthesize a confirm signal; if it fails to send the confirm (e.g., due to a network partition), the timer rolls back, potentially causing a flapping state where the change is applied and rolled back repeatedly.
• Limited rollback granularity – rollback restores the entire previous configuration, undoing unrelated, intentional changes made concurrently by another process.

Phased Groups in Risky Automation

Definition and Purpose

A phased group (wave, batch, or canary group) is a logical partition of the target device set that receives the configuration change in sequential, isolated steps. Each phase is treated as an independent transaction: automation waits for a verification gate to pass before advancing to the next phase. This limits blast radius, provides early fault detection, and enables controlled rollback of only the affected subset.

Implementing Phased Groups

1. Inventory segmentation – divide inventory into N groups based on geography, role, redundancy tier, etc. (e.g., group1 = edge‑routers‑us‑east, group2 = core‑routers‑us‑west).
2. Phase loop – for each group:
   • Push candidate configuration to all devices in the group (parallel transport: Nornir, Ansible, custom gNMI client).
   • Start a phase‑specific verification gate (see Independent Verification section).
   • If verification passes, issue an explicit confirm to make the change permanent on that group.
   • If verification fails, trigger a rollback limited to the devices in the current group only.
   • Optionally insert a manual approval or a “hold‑time” before proceeding to the next group.
3. State tracking – persist the phase index and confirmation status in a durable store (ConfigMap, database table, file) so a resumed run knows where to continue.

Example Code for Phased Groups (Python/Nornir)

# phased_rollout.py
from nornir import InitNornir
from nornir_netconf import netconf_commit_confirm, netconf_rollback
from nornir_utils.plugins.functions import print_result
from nornir.core.task import Result, Task
import time

# ------------------------------------------------------------------
# Configuration to push (as a string or template rendered elsewhere)
# ------------------------------------------------------------------
CANDIDATE_CFG = """
interface GigabitEthernet0/0/0
 description Uplink to ISP
 ip address 203.0.113.2 255.255.255.252
!
"""

# ------------------------------------------------------------------
# Verification function: check that the interface is up and has the expected IP
# ------------------------------------------------------------------
def verify_interface(task: Task) -> Result:
    from gnmi.client import gNMIClient
    try:
        with gNMIClient(
            target=f"{task.host.hostname}:57400",
            username=task.host.username,
            password=task.host.password,
            insecure=True,
        ) as client:
            resp = client.get(
                path=[
                    "interfaces/interface[name=GigabitEthernet0/0/0]/state/oper-status",
                    "interfaces/interface[name=GigabitEthernet0/0/0]/ipv4/address[ip=203.0.113.2]/state/ip",
                ],
                encoding="json_ietf",
            )[0]
        oper = resp["val"]
        ip_addr = resp[1]["val"]
        if oper == "up" and ip_addr == "203.0.113.2":
            return Result(host=task.host, result="PASS", failed=False)
        else:
            return Result(
                host=task.host,
                result=f"FAIL: oper={oper}, ip={ip_addr}",
                failed=True,
            )
    except Exception as e:
        return Result(host=task.host, result=f"EXC: {e}", failed=True)


# ------------------------------------------------------------------
# Main rollout logic
# ------------------------------------------------------------------
def phased_rollout(task: Task, phase: int, total_phases: int) -> Result:
    # 1. Load candidate config (netconf edit-config)
    task.run(
        task=netconf_commit_confirm,
        configuration=CANDIDATE_CFG,
        confirm_timeout=120,  # seconds
    )

    # 2. Run verification (independent of the NOS timer)
    verify_result = task.run(task=verify_interface)
    if verify_result.failed:
        # 3a. On failure, rollback only this device
        task.run(task=netconf_rollback)
        return Result(
            host=task.host,
            result=f"Phase {phase}: Verification failed – rolled back",
            failed=True,
        )
    else:
        # 3b. On success, send explicit confirm to make permanent
        task.run(task=netconf_commit_confirm, confirm=False)  # confirm only
        return Result(
            host=task.host,
            result=f"Phase {phase}: Verified and confirmed",
            failed=False,
        )


def main():
    nr = InitNornir(config_file="config.yaml")
    # Example: split inventory into two phases by site
    phase1 = nr.filter(F(site__eq="us-east"))
    phase2 = nr.filter(F(site__eq="us-west"))

    for idx, group in enumerate([phase1, phase2], start=1):
        print(f"\n=== Starting Phase {idx} ({len(group.inventory.hosts)} devices) ===")
        agg = nr.run(task=phased_rollout, phase=idx, total_phases=2, num_workers=20)
        print_result(agg)

        # Abort if any device in the phase failed
        if any(r.failed for r in agg.values()):
            print(f"Phase {idx} encountered failures – stopping rollout.")
            break
        else:
            print(f"Phase {idx} completed successfully.")
            # Optional manual hold before next phase
            # input("Press Enter to continue to next phase...")


if __name__ == "__main__":
    main()

Key points illustrated

• The NOS‑internal commit confirm timer (120 s) protects against lock‑out per device.
• The verify_interface task is independent of that timer; it validates service‑level correctness before we ever send the explicit confirm.
• Rollback is invoked only for devices that failed verification, limiting blast radius to the current phase.
• The outer loop enforces phased progression; a failure in phase 1 stops the rollout before any devices in phase 2 see the change.

Independent Verification in Automation

Importance of Independent Verification

Independent verification is a post‑apply, pre‑confirm check that the change satisfies the intended network state outside the NOS’s internal commit‑confirm mechanism. It provides:

• Semantic assurance – confirms that the configuration yields the expected forwarding behavior, not just that the parser accepted it.
• Early fault detection – allows rollback before the change is made permanent, reducing the window of potential impact.
• Observable evidence – generates logs, metrics, or ticket updates that can be audited and used for compliance.
• Decoupling from NOS timers – verification can be arbitrarily complex (multiple probes, synthetic traffic, control‑plane checks) without being constrained by a fixed timeout.

Methods for Implementing Independent Verification

CLI Examples for Independent Verification

Junos (using show commands)

# Verify that BGP peer 10.0.0.2 is Established
show bgp neighbor 10.0.0.2 | match "State/PfxRcvd: Established"

# Verify that interface ge-0/0/1 is up and has the expected IP
show interfaces ge-0/0/1 terse | match "ge-0/0/1.*up.*198.51.100.5/24"

# Verify that a firewall filter is applied and has non-zero hit count
show firewall filter my-filter | match "packet-count"

Cisco IOS‑XE (using show and ping)

# Check OSPF neighbor state
show ip ospf neighbor | include 10.0.0.3|FULL

# Verify interface status and IP
show ip interface brief | include GigabitEthernet0/1
# Expected line: GigabitEthernet0/1    203.0.113.10   YES manual up  up

# Synthetic traffic test – ping a known reachable host via the new path
ping 203.0.113.20 repeat 5 timeout 2

Arista EOS (using show and mac address-table)

# Verify that the interface is up
show interfaces Ethernet1 | include line protocol is up

# Confirm that a static MAC is learned
show mac address-table address 0011.2233.4455

By combining commit‑confirm (per‑device lock‑out protection) with phased groups, independent verification, and scoped rollback boundaries, automation achieves true safety: blast radius is limited, faults are caught early, and rollbacks affect only the devices that actually need them. Relying solely on commit‑confirm timers leaves the network exposed to logical errors, cascading failures, and operator‑dependent flapping—precisely the myth we set out to debunk.