Emergency overrides without bypassing containment entirely

Introduction to Break-Glass Paths

Definition and Purpose

A break‑glass path is a deliberately constrained, high‑priority change mechanism that allows operators to bypass normal change‑control gates when an urgent operational issue threatens service availability, security, or safety. Unlike a full “emergency override”, a break‑glass path preserves a minimal set of safeguards—scoped execution, immutable logging, and a verifiable rollback trigger—so that speed does not completely sacrifice accountability.

Benefits and Trade‑Offs

Designing Break-Glass Paths

Identifying Urgent Change Scenarios

Urgent changes typically fall into one of the following categories:

1. Security patches for actively exploited vulnerabilities (e.g., remote code execution).
2. Network‑level mitigations such as BGP blackholing, ACL tightening, or QoS re‑prioritization during a DDoS attack.
3. Configuration roll‑backs to a known‑good state after a faulty push that caused service degradation.
4. Resource reclamation (e.g., emergency deletion of a runaway container or VM) to prevent resource exhaustion.
5. Fail‑over initiation when automated health‑checks fail and manual intervention is required to avoid split‑brain.

Each scenario must be documented with:

• Trigger condition (observable metric, alarm, or threat intel).
• Maximum allowable execution time (e.g., ≤ 5 min from trigger to completion).
• Required approval level (often a single on‑call engineer with a secondary verbal confirmation).

Minimal Scoping Considerations

The goal is to limit the blast radius to the smallest set of devices, services, or data planes that can affect the symptom. Scoping steps:

1. Impact mapping – use configuration‑management databases (CMDB) or service‑dependency graphs to identify all nodes that directly influence the failing component.
2. Atomic unit selection – choose the smallest configurable unit (e.g., a single interface, a specific VRF, a particular security‑group rule) that, when changed, addresses the trigger.
3. Exclusion list – explicitly list any adjacent systems that must remain untouched (e.g., peer routers, shared storage clusters).
4. Dry‑run validation – run the intended change in a read‑only or simulation mode (if the platform supports it) to confirm no unintended side‑effects.

If the scoped set cannot be reduced below a predefined threshold (e.g., > 5 % of total devices or > 10 % of traffic), the break‑glass request must be escalated to a full emergency change process.

Logging and Auditing Requirements

Every break‑glass execution must generate:

• Immutable log entry in a write‑once storage system (e.g., WORM‑enabled S3 bucket, Cloud Audit Logs, or a signed journal).
• Structured fields: timestamp, initiator identity, justification text, scoped target list, exact CLI/API payload, pre‑check results, verification gate outcome, and rollback trigger condition.
• Cryptographic chaining – each entry includes a hash of the previous entry to detect tampering.
• Retention policy – minimum 12 months for compliance (e.g., PCI‑DSS, ISO 27001) and longer for critical infrastructure sectors.
• Access control – read‑only for auditors; write access limited to the break‑glass automation service and privileged operators.

Rollback Control Mechanisms

A break‑glass path must define a rollback trigger that is evaluated after the verification gate. Typical triggers:

• Health‑check failure (e.g., loss of connectivity, increased error rate > threshold).
• Manual abort – operator issues a breakglass abort command within the rollback window.
• Time‑based expiry – if verification does not succeed within a predefined window (e.g., 2 min), the system auto‑initiates rollback.

Rollback implementation options:

The chosen mechanism must be tested in a non‑production environment to confirm that it restores both configuration and any transient state (e.g., session tables, QoS counters).

Implementing Break-Glass Paths

Code Examples for Automated Workflows

Below is a Python‑based break‑glass runner that enforces the transaction boundaries described earlier. It uses Netmiko for device interaction, writes to an immutable log (AWS S3 with Object Lock), and performs a verification gate via a simple HTTP health‑check.

#!/usr/bin/env python3
import json, time, hashlib, boto3, sys, requests
from netmiko import ConnectHandler
from botocore.exceptions import ClientError

# -------------------- CONFIG --------------------
DEVICE = {
    "device_type": "cisco_ios",
    "host": "10.0.0.1",
    "username": "admin",
    "password": "*****",
    "secret": "*****",
}
LOG_BUCKET = "breakglass-logs"
LOG_KEY_PREFIX = "bg/"
VERIFICATION_URL = "https://health.example.com/status"
ROLLBACK_WINDOW = 120  # seconds
# ------------------------------------------------

def log_action(action: dict):
    """Write an immutable, chained log entry to S3."""
    s3 = boto3.client("s3")
    try:
        resp = s3.get_object(Bucket=LOG_BUCKET, Key=LOG_KEY_PREFIX + "latest")
        prev_hash = resp["Body"].read().decode().strip()
    except ClientError as e:
        if e.response["Error"]["Code"] == "NoSuchKey":
            prev_hash = "0" * 64  # genesis
        else:
            raise

    action["prev_hash"] = prev_hash
    action["timestamp"] = time.time()
    action_json = json.dumps(action, sort_keys=True)
    action_hash = hashlib.sha256(action_json.encode()).hexdigest()
    action["entry_hash"] = action_hash

    log_entry = action_json + "\n"
    s3.put_object(
        Bucket=LOG_BUCKET,
        Key=LOG_KEY_PREFIX + f"{int(time.time())}.log",
        Body=log_entry.encode(),
        ObjectLockMode="GOVERNANCE",
        ObjectLockRetainUntilDate=int(time.time()) + 31536000,  # 1 year
    )
    s3.put_object(
        Bucket=LOG_BUCKET,
        Key=LOG_KEY_PREFIX + "latest",
        Body=action_hash.encode(),
    )

def run_prechecks(conn):
    """Example: ensure target‑specific pre‑checks."""
    conn.send_command("show version", expect_string=r"#")
    out = conn.send_command("show access-lists | include BLOCK-DDOS")
    if "BLOCK-DDOS" in out:
        raise RuntimeError("Conflicting ACL already present")
    return True

def apply_change(conn):
    """Apply the scoped change – create a temporary ACL."""
    cmds = [
        "ip access-list extended BLOCK-DDOS",
        " deny ip any any log",
        " permit ip any any",
        "exit",
        "interface GigabitEthernet0/0",
        " ip access-group BLOCK-DDOS in",
        "exit",
    ]
    conn.send_config_set(cmds)

def verify_change():
    """Simple HTTP‑based health check."""
    try:
        r = requests.get(VERIFICATION_URL, timeout=5)
        return r.status_code == 200 and r.json().get("ok") is True
    except Exception:
        return False

def rollback(conn):
    """Remove the temporary ACL."""
    cmds = [
        "no ip access-group BLOCK-DDOS in",
        "interface GigabitEthernet0/0",
        "no ip access-group BLOCK-DDOS in",
        "exit",
        "no ip access-list extended BLOCK-DDOS",
    ]
    conn.send_config_set(cmds)

def main(justification):
    conn = ConnectHandler(**DEVICE)
    conn.enable()
    try:
        # ---- PRECONDITIONS ----
        if not run_prechecks(conn):
            sys.exit(1)

        # ---- ACTION LOGGING (pre‑apply) ----
        log_action({
            "actor": "breakglass_operator",
            "justification": justification,
            "phase": "precheck",
            "device": DEVICE["host"],
            "scoped_targets": ["GigabitEthernet0/0"],
        })

        # ---- CHANGE APPLY ----
        apply_change(conn)

        # ---- ACTION