When a Diagnostic Agent Must Stop and Escalate

Introduction to Auto‑Remediation and Root Cause Analysis

Auto‑remediation agents observe telemetry, infer a probable fault, and apply corrective actions without human intervention. In network and infrastructure contexts they typically:

1. Detect an anomaly (e.g., interface flap, CPU spike, BGP session loss).
2. Hypothesize a root cause using rule‑based reasoning, correlation, or lightweight ML models.
3. Act by pushing a remediation payload (CLI, NETCONF/RESTCONF, API call) intended to restore service.

The agent operates inside a safe time window (STW) – a pre‑defined interval during which it may attempt auto‑remediation before the risk of uncontrolled change outweighs the benefit. If the agent cannot prove the hypothesized root cause within the STW, it must cease further autonomous action and hand control to operators.

Root cause analysis (RCA) provides the evidentiary basis that a remediation action addresses the underlying fault rather than merely suppressing symptoms. Without proven RCA:

• Mis‑directed changes can exacerbate the problem (e.g., disabling a healthy link while the real issue is a mis‑configured ACL).
• Rollback uncertainty increases because the inverse operation may not be known or safe.
• Operator trust erodes, leading to manual overrides that bypass safety controls.

Consequently, the design of stop conditions, escalation rules, and operator intervention points must be anchored to the agent’s ability to prove causality within the STW.

Stop Conditions for Auto‑Remediation

Time‑Based Stop Conditions

A hard timeout is the most deterministic guard.

If elapsed_time > max_attempt_duration or attempt_count >= total_allowed_attempts, the agent must stop further auto‑remediation and trigger the escalation path.

Threshold‑Based Stop Conditions

Telemetry‑driven thresholds stop the agent when the symptom persists or worsens despite remediation.

• Symptom Persistence Threshold – If the observed metric (e.g., packet loss %) remains above symptom_threshold for two consecutive evaluation cycles after a remediation attempt, stop.
• Degradation Threshold – If a secondary metric (e.g., CPU utilization) rises > degradation_delta (e.g., +15 %) after remediation, abort – the action may have introduced a new problem.
• Error‑Rate Threshold – If the agent logs > max_internal_errors (e.g., 5) validation or communication failures within the STW, cease auto‑remediation to avoid cascading faults.

These thresholds are evaluated after each remediation attempt; they are post‑action validation gates.

External Intervention Stop Conditions

External signals can override internal logic:

• Operator‑Issued Pause – CLI command (agent pause) or API flag (pause=true) set by an NOC analyst.
• Maintenance Window – If a scheduled change window begins, the agent must honor a maintenance_mode flag and suspend autonomous actions.
• Higher‑Priority Alarm – Reception of a severity‑critical event (e.g., power loss) from an external EMS triggers an immediate stop, regardless of internal state.

Implementation: the agent subscribes to a lightweight pub/sub topic (e.g., MQTT agent/control) and checks the payload on each evaluation loop. A STOP payload forces an immediate exit from the remediation routine.

Escalation Rules for Unresolved Issues

Escalation Triggers

An escalation is launched when any stop condition fires and the agent cannot prove root cause. Triggers include:

1. Timeout Expiry – max_attempt_duration exceeded.
2. Threshold Violation – Symptom persists or degradation observed post‑remediation.
3. External Stop – Operator pause, maintenance window, or higher‑priority alarm.
4. Internal Fault – Repeated validation/communication errors (> max_internal_errors).

Each trigger carries a severity tag (low, medium, high, critical) derived from the original symptom’s impact classification.

Escalation Levels and Notifications

Notifications contain a standard payload:

{
  "agent_id": "agent-01",
  "timestamp": "2025-11-03T14:22:07Z",
  "trigger": "timeout",
  "symptom": {
    "type": "bgp_session_down",
    "peer": "10.0.1.5",
    "duration_s": 120
  },
  "attempts": 2,
  "last_action": {
    "method": "netconf",
    "payload": "<config><bgp><neighbor><peer-address>10.0.1.5</peer-address><enabled>false</enabled></neighbor></bgp></config>"
  },
  "rca_status": "unproven",
  "recommended_next": "manual investigation"
}

Escalation Response and Resolution

Upon receipt, the responder follows a runbook that mandates:

1. Acknowledge the alert within the SLA (timestamped acknowledgment).
2. Collect raw telemetry (interface counters, logs, SNMP walks) for the affected entity.
3. Validate the agent’s hypothesis by attempting a manual remediation (if safe) or gathering further evidence.
4. Document findings in the incident ticket, marking the RCA status as proven, unproven, or inconclusive.
5. Decide on next steps:
   • If RCA proven → apply permanent fix, close ticket.
   • If RCA unproven but symptom mitigated → observe for recurrence, schedule a post‑mortem.
   • If symptom worsens → invoke higher‑level escalation (L2→L3) or initiate a network‑wide safety hold.

All actions are logged to an immutable audit store (e.g., WORM‑enabled S3 bucket) to satisfy compliance and facilitate post‑incident analysis.

Operator Intervention Points

Identification of Intervention Points

Intervention points are deliberately placed before the agent commits a change that cannot be safely reverted automatically.

If the agent cannot prove root cause before P1, it must halt at P1 and escalate.

Operator Notification and Alerting

At each intervention point, the agent emits a structured event to the ops bus:

• P1 – agent/plan/proposed
• P2 – agent/plan/dryrun (includes dry‑run result)
• P3 – agent/change/applied (includes config diff)
• P4 – agent/change/observed (includes metric snapshot)

Operators subscribe to these topics via a monitoring platform (e.g., Grafana Alerting, Splunk). Alerts include a deep link to the device’s live CLI/snapshot view, enabling rapid inspection.

Operator Intervention Procedures

Standard Operating Procedure (SOP) for a manual intervention at P3 (change application) is:

1. Receive agent/change/applied event.
2. SSH or NETCONF to the target device.
3. Verify the applied diff matches the intended remediation (show config | compare rollback 0).
4. Check key health indicators (interface state, protocol adjacency, CPU).
5. If health is good → acknowledge and close the auto‑remediation ticket.
6. If health is degraded → issue the rollback command (if the platform supports atomic rollback, e.g., rollback 0 on Juniper, configure replace rollback 1 on Cisco IOS XR) or apply a known‑good baseline config.
7. Document the manual action in the incident ticket, marking the RCA status as unproven (since the agent could not prove cause).

All CLI commands used in the SOP are logged automatically via the device’s AAA accounting.

Troubleshooting Unsuccessful Auto‑Remediation

Common Issues and Errors

Debugging and Logging Techniques

• Structured Logging – Emit JSON logs with fields: timestamp, agent_id, event_type, payload_hash, return_code. Enable log_level=debug during troubleshooting.
• Packet Capture – Use tcpdump -i any -s 0 -w agent.pcap port 830 or port 161 to capture NETCONF/SNMP exchanges.
• Trace IDs – Propagate a UUID (X-Request-ID) through all agent‑device interactions; correlate logs on both sides.
• Metric Snapshots – Before and after each remediation, store a JSON snapshot of relevant MIBs (ifInOctets, bgpPeerState, cpuUtilization).
• Health‑Check Endpoint – Expose /health HTTP endpoint returning {status:"ok", last_success:<ts>, pending_attempts:<n>} for external monitoring.

Code Examples for Troubleshooting

Bash snippet to capture NETCONF traffic and validate a dry‑run:

#!/usr/bin/env bash
set -euo pipefail

AGENT_ID="agent-01"
DEVICE="10.0.1.5"
PORT=830
OUTDIR="/var/log/agent/traces"
mkdir -p "$OUTDIR"
TS=$(date +%s)
PCAP="${OUTDIR}/${AGENT_ID}_${DEVICE}_${TS}.pcap"

# Start background capture
tcpdump -i any -s 0 -w "$PCAP" port "$PORT" &
TCPDUMP_PID=$!

# Invoke agent dry-run via its CLI (assuming it exposes a subcommand)
agent-cli --device "$DEVICE" --dry-run --output json > "${OUTDIR}/${AGENT_ID}_${DEVICE}_${TS}_dryrun.json"
DRYRUN_EXIT=$?

# Stop capture
kill "$TCPDUMP_PID"
wait "$TCPDUMP_PID" 2>/dev/null

if [[ $DRYRUN_EXIT -ne 0 ]]; then
    echo "Dry‑run failed; see ${OUTDIR}/${AGENT_ID}_${DEVICE}_${TS}_dryrun.json"
    exit 1
fi

echo "Dry‑run succeeded. Capture stored at $PCAP"

Python helper to compare pre/post metric snapshots:

import json
from pathlib import Path

def load_snapshot(path: Path) -> dict:
    return json.loads(path.read_text())

def compare_snapshots(before: dict, after: dict, keys: list) -> dict:
    diff = {}
    for k in keys:
        b = before.get(k)
        a = after.get(k)
        if b != a:
            diff[k] = {"before": b, "after": a}
    return diff

# Example usage
before = load_snapshot(Path("metrics_before.json"))
after  = load_snapshot(Path("metrics_after.json"))
relevant_keys = ["ifInOctets", "bgpPeerState", "cpuUtilization"]
changes = compare_snapshots(before, after, relevant_keys)
print(json.dumps(changes, indent=2))

End of document.