Designing a Safe Operating Model for an LLM‑Driven Network Remediation Loop

Author: Sophia Lin – Cloud & Automation Architect
Date: 2025‑09‑24

Introduction

Large Language Models (LLMs) can interpret alarms, query telemetry, and invoke external tools via function‑calling interfaces. In network operations this enables the LLM to propose and execute remediation actions such as interface shutdowns, ACL updates, or BGP peer resets. The same capabilities introduce risk vectors: tool misuse, hallucinated evidence, prompt injection, opacity, and cascading failure.

A Safe Operating Model (SOM) defines boundaries, evidence standards, human‑in‑the‑loop checks, and rollback guarantees that allow the LLM to act as a trusted assistant rather than an autonomous agent. The SOM must:

1. Limit the blast radius of any action to a pre‑approved scope.
2. Require verifiable evidence before issuing state‑changing commands.
3. Present clear confirmation prompts that enable an operator to veto or modify the plan.
4. Escalate to a human when uncertainty, conflict, or policy violation is detected.
5. Provide immutable audit trails for every LLM‑tool interaction.

The following sections detail a concrete SOM that satisfies these goals while remaining grounded in current tooling and model capabilities.

Action Scopes and Boundaries

Permitted Network Tools and Actions

We start with an allow‑list of network primitives the LLM may invoke. Each primitive maps to a concrete tool (CLI, NETCONF/RESTCONF endpoint, or automation playbook) and is annotated with resource type, operation class, and maximum impact.

Any action not present in this matrix is implicitly denied. The list is version‑controlled and reviewed quarterly by the NetSec team.

Action Scope Hierarchy and Inheritance

Actions are organized in a tree that mirrors administrative domains:

Network
├── Core
│   ├── Router‑A
│   │   ├── Interface‑Eth1/1
│   │   └── ACL‑INET
│   └── Router‑B
├── Distribution
│   └── Switch‑C
└── Access
    ├── AP‑Group‑1
    └── VLAN‑10

• Inheritance rule: A role granted at a parent node automatically inherits all child‑node actions unless overridden by an explicit deny.
• Override mechanism: A JSON policy file can list deny entries that prune inheritance (e.g., deny shutdown on Core routers for the NOC‑Junior role).

This hierarchy enables coarse‑grained role definitions while still allowing fine‑grained exceptions.

Role‑Based Access Control for the LLM

The LLM runs under a service account (llm-remediator) mapped to a set of RBAC roles. Enforcement occurs at the tool‑calling gateway:

1. Receives a tool‑call request from the LLM (JSON‑RPC over HTTPS).
2. Looks up the caller’s service account → RBAC profile.
3. Checks the requested action against the allow‑list and any deny overrides.
4. Returns 403 Forbidden if the check fails; otherwise forwards the call to the underlying device or automation engine.

All decisions are logged with timestamp, service_account, action, resource, decision, reason. The gateway runs in a hardened container with a read‑only rootfs, and its policy store is signed and verified at startup.

Evidence Requirements and Collection

Types of Evidence for Remediation Loop Decisions

To move from proposal to execution the LLM must present corroborating evidence satisfying one of the following categories:

At least two independent evidence types are required for a mutating action; read‑only actions may proceed with a single evidence piece.

Evidence Collection Mechanisms

The Evidence Collector runs as a sidecar to the LLM gateway and exposes a REST endpoint /evidence?resource=<uri>&types=<list> returning a JSON payload:

{
  "resource": "interface:Eth1/1",
  "evidence": {
    "telemetry": {"ifInErrors": 1245, "timestamp": "2025-09-24T10:12:00Z"},
    "log": ["%LINK-3-UPDOWN: Interface Ethernet1/1, changed state to down"],
    "config_diff": ["- shutdown", "+ no shutdown"]
  }
}

Data Quality and Validation Checks

Before evidence is handed to the LLM, the collector performs:

1. Schema validation – each evidence block must conform to a JSON‑Schema (numeric fields, ISO‑8601 timestamps).
2. Freshness check – telemetry older than max_age (default 60 s) is flagged stale.
3. Cross‑source consistency – conflicting signals (e.g., log indicates down but telemetry shows zero errors) raise a conflict flag.
4. Signature verification – for config snapshots, a SHA‑256 hash signed by the config‑repo key is verified.

If any check fails, the evidence is annotated with a quality field (good, stale, conflict, invalid). The LLM treats stale or conflict as insufficient for mutating actions and requests additional evidence or escalates.

Confirmation Prompts and User Interaction

Designing Effective Confirmation Prompts

A confirmation prompt must convey:

• What the LLM intends to do (action, target, parameters).
• Why it believes the action is warranted (evidence summary).
• What the possible impact is (blast radius, rollback path).
• How the operator can respond (approve, modify, reject, request more info).

We use a structured markdown template rendered in the operator’s UI (Open WebUI chat pane, Slack bot, etc.):

## ⚠️ Proposed Remediation
**Action:** `shutdown interface Ethernet1/1`  
**Target:** Router‑A (core)  
**Reason:**  
- Telemetry: ifInErrors 1245/min (threshold 1000/min)  
- Log: `%LINK-3-UPDOWN: Interface Ethernet1/1, changed state to down`  
**Impact:**  
- Immediate loss of L2/L3 connectivity for VLAN 10 (≈ 200 hosts)  
- Rollback: `no shutdown interface Ethernet1/1` (instant)  
**Evidence Quality:** telemetry = good, log = good  

Please respond with one of:  
`approve` – execute as described  
`modify <new‑params>` – e.g., `modify shutdown interface Ethernet1/2`  
`reject` – abort and escalate to human  
`more` – request additional evidence

The prompt is versioned (prompt-v1.2) and stored in the policy repo; any change requires peer review.

User Interaction Modes

All modes funnel through the same Authorization Service, which requires an explicit approve token signed by the operator’s LDAP/Kerberos principal before a mutating action proceeds.

Prompt Engineering for Safe Interactions

• Temperature set to 0.0 for deterministic tool selection; the final natural‑language response uses a low temperature (0.2) to allow phrasing variation.
• System message includes the SOM policy excerpt: “You may only propose actions that appear in the allow‑list. You must cite at least two evidence types. You must never assume device state without evidence.”
• Few‑shot examples of correct and incorrect prompts are embedded in the model’s context window (~800 tokens) to reduce hallucination.
• Self‑check: before emitting the final prompt, the LLM runs an internal validation step (via a separate “verifier” model) that scans for missing evidence or disallowed action tokens; if the verifier flags an issue, the LLM revises its output up to three times.

These techniques have empirically reduced unsafe proposals by > 80 % in internal red‑team exercises.

Escalation Conditions and Human Intervention

Escalation Triggers

Escalation is automatic when any condition evaluates to true:

When a trigger fires, the system halts the remediation loop, logs the reason, and initiates the escalation workflow.

Conditions Requiring Human Intervention

Human intervention is mandatory for:

• Any escalation triggered above.
• Change‑window actions (e.g., modifications during a regulated maintenance window) that require Change Advisory Board (CAB) approval per ITIL.
• Security‑sensitive resources (e.g., firewall policy, RADIUS secrets) irrespective of confidence.

The human receives a structured escalation packet containing:

• Original ticket/alarm.
• LLM’s proposed action and evidence.
• Escalation reason(s) with timestamps.
• Suggested manual remediation steps (generated by a separate “runbook‑retriever” model).
• Links to relevant dashboards (Grafana, NetBox).

The operator can then:

• Approve the LLM proposal (overriding the escalation).
• Modify the proposal (e.g., change target interface).
• Reject and execute a manual procedure.
• Defer to a specialist team (e.g., security).

All decisions are recorded with the operator’s identity and a cryptographic signature (using their PGP key or SSH certificate) for non‑repudiation.

Escalation Protocols and Communication

1. Internal alert – The gateway publishes an event to a Kafka topic llm.escalations.
2. Notification – A consumer service forwards the alert to:
   • PagerDuty (high‑severity)
   • Email/SMS distribution lists (medium‑severity)
   • A dedicated Slack channel for triage (low‑severity)

The alert includes the escalation packet JSON, enabling automated ticket creation in ITSM systems. Operators acknowledge via the same channels, and their response is cryptographically signed and appended to the audit trail.

End of document.