Controller Outage Approval Modes

Introduction to Autonomy in System Failure

Definition of Autonomy in System Context

Autonomy, in the context of network and infrastructure operations, refers to the capability of a system to perceive its state, make decisions, and execute actions without continuous human intervention, while adhering to predefined policy boundaries. An autonomous component typically combines sensing (telemetry, health checks), reasoning (policy evaluation, risk assessment), and actuation (configuration changes, service restarts) within a closed-loop control architecture.

Importance of Autonomy During Partial Failures

When a subset of controllers or inventory data becomes unavailable, the operational surface shrinks but critical services often must continue. Autonomy can:

• Preserve forwarding‑plane stability by locally enforcing last‑known‑good configurations.
• Prevent cascading misconfigurations by refusing to act on stale or contradictory data.
• Buy time for human operators to restore degraded control paths without causing service outages.
  However, autonomy must be bounded; unchecked local actions can diverge from the intended network intent and create hidden debt.

Understanding Partial Controller or Inventory Failure

Types of Partial Failures

Impact of Partial Failures on System Operations

• Decision latency: The autonomy engine may need to wait for time‑outs before falling back to local policies, increasing reaction time from sub‑second to several seconds.
• State divergence: Devices that remain connected may receive new intent while isolated devices retain old configuration, leading to transient loops or blackholes.
• Approval bottlenecks: If approval paths rely on a central ticketing system that is partially unavailable, change requests may queue or be silently dropped.
• Rollback impairment: Without a reliable rollback channel, a failed change cannot be safely reverted, increasing blast radius.

Approval Paths and Their Role in Autonomy

Definition and Function of Approval Paths

An approval path is the structured workflow that validates a proposed change against business, security, and operational policies before it is applied. Typical steps include:

1. Change request creation (via CLI, GUI, or API).
2. Policy evaluation (e.g., change‑window, impact analysis, peer review).
3. Authorized sign‑off (human or automated approver).
4. Enqueue for execution (often via a change‑management system like ServiceNow, Jira, or a custom webhook).

In autonomous systems, the approval path may be fully automated (policy‑as‑code) or semi‑automated (human‑in‑the‑loop for high‑risk changes).

Consequences of Approval Path Degradation

When the approval path is degraded:

• False positives: Legitimate changes may be blocked, causing operational stagnation.
• False negatives: Unreviewed changes may slip through if the system defaults to “allow on failure” to avoid deadlock.
• Audit gaps: Missing approval records break compliance trails, complicating post‑incident analysis.
• Trust erosion: Operators may bypass the path entirely, undermining the governance model.

Source of Truth and Its Significance

Definition and Importance of Source of Truth

The Source of Truth (SoT) is the authoritative repository that stores the desired state of the network: device inventory, intended configuration, topology, and policy bindings. In modern architectures this is often a Git repository (NetOps/GitOps) or a distributed key‑value store (etcd, Consul) synchronized with the controller.

The SoT enables:

• Idempotent reconciliation: Controllers continuously compare running state vs. SoT and drift‑correct.
• Change traceability: Every commit corresponds to a known intent change.
• Rollback basis: Reverting a commit restores the previous SoT snapshot.

Effects of Source of Truth Degradation on Autonomy

If the SoT is partially unavailable or corrupted:

• Reconciliation loops: Controllers may repeatedly attempt to apply missing or conflicting intents, generating churn.
• Drift acceptance: Autonomy may fall back to “running‑config is correct” mode, silently persisting undesired state.
• Risk of divergent intents: Different controller instances may derive conflicting configurations from fragmented SoT views, leading to protocol mismatches.
• Recovery complexity: Rebuilding the SoT from device snapshots is costly and may miss transient intent (e.g., dynamic ACLs).

Rollback Channels and Their Purpose

Definition and Function of Rollback Channels

A rollback channel is the mechanism by which a previously applied change can be reversed, restoring the system to a known good state. It typically comprises:

• Versioned state storage (e.g., Git commit history, snapshot database).
• Inverse operation generation (e.g., no form of CLI commands, DELETE in RESTCONF).
• Execution pathway (orchestrator, Ansible playbook, NETCONF <edit-config> with operation="remove").
• Verification step (post‑rollback validation via telemetry or compliance checks).

Consequences of Rollback Channel Degradation

When rollback channels are degraded:

• Irreversible changes: A faulty push may remain until a manual window, extending impact.
• Increased mean‑time‑to‑repair (MTTR): Operators must craft ad‑hoc reversal scripts, raising error probability.
• Cascade risk: Without reliable rollback, operators may avoid necessary changes, leading to technical debt accumulation.
• Compliance violation: Inability to demonstrate rollback capability can breach SLAs or regulatory requirements.

Autonomy Response to Simultaneous Degradation

Initial Assessment and Prioritization

When approval paths, SoT, and rollback channels are all impaired, the autonomy engine must:

1. Detect degradation via health‑check endpoints (e.g., /ready on controller, inventory DB lag metrics, approval‑service heartbeat).
2. Classify failure scope (local vs. global) using quorum checks (e.g., etcd member list, consensus leader status).
3. Prioritize actions:
   • Safety first: Preserve data‑plane forwarding and prevent loops.
   • Minimize new risk: Defer non‑essential changes.
   • Enable manual intervention: Expose degraded state via alerts and runbooks.

A typical decision tree (pseudo‑code) might look like:

def assess_degradation():
    controller_ok = ping_controller() and check_controller_health()
    sot_ok      = check_inventory_lag() < MAX_LAG and check_inventory_quorum()
    approval_ok = check_approval_service_heartbeat()
    rollback_ok = check_rollback_store_availability()

    if not controller_ok:
        return "CONTROLLER_DOWN"
    if not sot_ok:
        return "SOT_DEGRADED"
    if not approval_ok:
        return "APPROVAL_DEGRADED"
    if not rollback_ok:
        return "ROLLBACK_DEGRADED"
    return "HEALTHY"

Temporary Workarounds and Contingency Plans

Code Examples for Automated Contingency Measures

Below is a Python snippet that implements a local‑cache fallback for the SoT when the inventory service lag exceeds a threshold. It assumes each controller runs a side‑car process that periodically snapshots the inventory etcd store to a local RocksDB instance.

import time
import rocksdb
import json
from grpc import insecure_channel
from inventory_pb2 import GetDeviceRequest
from inventory_pb2_grpc import InventoryServiceStub

INVENTORY_LAG_SEC = 5
CACHE_DB_PATH = "/var/lib/autonomy/sot_cache.db"

def _open_cache():
    return rocksdb.DB(CACHE_DB_PATH, rocksdb.Options(create_if_missing=True))

def get_device_from_cache(device_id: str):
    db = _open_cache()
    raw = db.get(device_id.encode())
    db.close()
    if raw is None:
        raise KeyError(f"Device {device_id} not in cache")
    return raw.decode()

def get_device_with_fallback(device_id: str) -> dict:
    # Try live inventory first
    try:
        channel = insecure_channel("inventory-svc:50051")
        stub = InventoryServiceStub(channel)
        resp = stub.GetDevice(GetDeviceRequest(device_id=device_id), timeout=2)
        # Update cache on success
        db = _open_cache()
        db.put(device_id.encode(), resp.json.encode())
        db.close()
        return resp.json
    except Exception as e:
        # Live call failed or timed out – check lag
        lag = get_inventory_replication_lag()  # external metric fetch
        if lag > INVENTORY_LAG_SEC:
            # Fallback to cache
            try:
                return json.loads(get_device_from_cache(device_id))
            except KeyError:
                raise RuntimeError(
                    f"Both live inventory and cache unavailable for {device_id}"
                ) from e
        else:
            # Transient glitch – retry shortly
            time.sleep(0.5)
            return get_device_with_fallback(device_id)

Explanation of safety bounds:

• The fallback is read‑only; no writes are attempted to the cache during degradation.
• A max‑lag threshold prevents using a stale cache when the inventory is merely slow but still consistent.
• The function raises an explicit error if neither source can satisfy the request, forcing the autonomy engine to enter a safe‑state (e.g., hold current config, raise alert).

Troubleshooting Strategies for Autonomy

Identifying Root Causes of Failure

A systematic approach combines layered health checks, consensus diagnostics, and telemetry correlation:

1. Controller health – curl -s http://controller:8080/ready → 200/503.
2. Inventory quorum – etcdctl endpoint status --cluster -w table.
3. Approval service – systemctl show approval-svc --property=ActiveState.
4. Rollback store – ls -l /var/rollback/snapshots/*.json | wc -l.
5. Data‑plane impact – show ip route | grep <affected-prefix> on representative routers.

If multiple layers fail simultaneously, the common dependency (e.g., network partition, shared storage) is suspect.

CLI Commands for Diagnostic Purposes

Example Use Cases for Troubleshooting Tools

• Scenario: Approval service appears unresponsive, but controller and inventory are healthy.
  • Tool: curl -I http://approval-svc:8080/api/v1/approvals returns 502 Bad Gateway.
  • Action: Inspect reverse