A safe workbench from PromQL to packet capture

Introduction to Operator Workbench Design

Overview of Requirements

The operator workbench enables a network engineer to move fluidly from a Prometheus‑derived symptom (e.g., a spike in interface error counters or latency SLO breach) to targeted packet capture and device‑level inspection without granting automated scripts unrestricted write or reconfiguration privileges.

Functional requirements

1. Evidence ingestion – consume Prometheus alert payloads (via Alertmanager webhook or direct query) and surface the relevant metric series, labels, and timestamps in a unified UI.
2. Pivot trigger – a single operator action (click, command, or approved API call) that launches a packet‑capture job on the implicated interface or device, preserving the original alert context.
3. Device inspection – after capture, provide read‑only access to CLI‑style show commands, log tailing, and safe‑mode diagnostics (e.g., show interface transceiver details, ping, traceroute).
4. Boundary enforcement – automation may only gather data; any change to device configuration must require explicit operator approval and be scoped to a predefined rollback window.
5. Auditability – every pivot, capture start/stop, and inspection command must be logged with operator identity, timestamp, and justification.

Non‑functional requirements

• Latency < 2 s from alert to capture start.
• Horizontal scalability to thousands of interfaces.
• Compliance with data‑protection regimes (GDPR, NERC CIP, etc.).

Importance of Trust Boundaries and Rollback Points

Trust boundaries delineate where automated agents may act without human consent. In this workbench the boundary is read‑only data collection. Crossing it (e.g., issuing configure terminal or write memory) must be gated by an explicit approval step and accompanied by a rollback point—a known‑good configuration snapshot or a change‑record that can be reverted automatically if post‑change validation fails.

Without such boundaries, an over‑aggressive AI‑driven pivot could:

• Trigger unintended configuration pushes that cause service disruption.
• Exfiltrate packet data beyond the scoped interface, violating privacy.
• Exhaust device CPU by launching uncontrolled capture sessions.

Rollback points provide a safety net: if the operator approves a change (e.g., applying a temporary ACL to isolate a faulty flow), the workbench records the pre‑change state and can invoke a rollback after a configurable validation window (e.g., 5 min of clean traffic) or on manual abort.

Architecture of Operator Workbench

High‑Level Components

+-------------------+      +-------------------+      +-------------------+
|   Alertmanager    | ---> |   Pivot Engine    | ---> |  Capture Orchestrator|
+-------------------+      +-------------------+      +-------------------+
          ^                         |                         |
          |                         v                         v
+-------------------+      +-------------------+      +-------------------+
|   Prometheus TSDB |      |   UI / Workbench  |      |   Device Agent    |
+-------------------+      +-------------------+      +-------------------+

• Alertmanager – receives alerts from Prometheus, enriches them with annotations (e.g., runbook_url, owner_team), and forwards a JSON payload to the Pivot Engine via a secure HTTPS webhook.
• Pivot Engine – a stateless service (Go/Rust) that validates the alert, enriches it with topology data (from a CMDB or NetBox), and presents options to the operator through the UI. It enforces that only read‑only actions are offered unless an approval gate is explicitly triggered.
• UI / Workbench – a single‑page application (React + TypeScript) running behind an internal auth proxy (OIDC or LDAP). It displays the alert context, provides a “Start Capture” button, and streams live CLI output via a WebSocket to a backend agent.
• Capture Orchestrator – translates the operator’s intent into device‑specific capture commands (e.g., tcpdump -i eth0 -w /var/captures/<alert-id>.pcap) and streams the resulting file to an object store (MinIO/S3) for later analysis. It also issues show commands via NETCONF/RESTCONF or SSH, depending on device capabilities.
• Device Agent – a lightweight daemon (Python/Go) residing on a management host or jump box that holds credential vaults (HashiCorp Vault, AWS Secrets Manager) and executes the privileged commands only after receiving a signed request from the Capture Orchestrator that includes an operator‑generated nonce and expiration timestamp.

All inter‑service communication uses mutual TLS (mTLS) and is logged to a centralized SIEM (e.g., Elasticsearch).

Integration with Prometheus

Prometheus continues to scrape metrics as usual. The workbench does not alter scrape intervals or add new exporters; it merely consumes the existing alert stream.

Alert payload example (JSON sent to Pivot Engine):

{
  "receiver": "team-netops",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "HighInterfaceErrors",
        "device": "router01",
        "iface": "GigabitEthernet0/1/0",
        "severity": "critical"
      },
      "annotations": {
        "summary": "Interface error rate > 100 eps for 2m",
        "runbook": "https://wiki.example.com/runbooks/high-iface-errors"
      },
      "startsAt": "2025-09-24T12:00:00Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus.example.com/graph?g0.expr=..."
    }
  ],
  "groupLabels": {},
  "commonLabels": {
    "alertname": "HighInterfaceErrors",
    "device": "router01",
    "iface": "GigabitEthernet0/1/0"
  },
  "commonAnnotations": {
    "summary": "Interface error rate > 100 eps for 2m",
    "runbook": "https://wiki.example.com/runbooks/high-iface-errors"
  }
}

The Pivot Engine validates the device and iface labels against the CMDB; if missing, it raises a “needs manual triage” flag and does not offer a capture button.

Packet Capture and Device Inspection Tools

All tools are invoked via the Device Agent, which drops privileges to a non‑root user after acquiring necessary capabilities (e.g., CAP_NET_RAW for tcpdump). The agent enforces a maximum capture duration (default 30 s) and a maximum file size (default 100 MiB) to prevent resource exhaustion.

Designing the Pivot Mechanism

Triggering Pivot from Prometheus Evidence

When the UI receives an alert payload, it renders a Pivot Card containing:

• Timestamp and severity.
• Key metric values (e.g., ifInErrors, ifOutDiscards).
• Links to the underlying Prometheus graph (generatorURL).
• A Start Capture button that is disabled unless:

1. The alert labels map to a known device/interface in the CMDB.
2. The operator’s role includes capture:init (checked via OIDC token claims).
3. No active capture exists for the same interface (to avoid duplicate sessions).

Clicking the button sends a signed POST request to the Pivot Engine:

POST /api/v1/pivot/capture
Authorization: Bearer <jwt>
Content-Type: application/json

{
  "alert_id": "router01_GigabitEthernet0/1/0_20250924T120000",
  "device": "router01",
  "iface": "GigabitEthernet0/1/0",
  "capture_opts": {
    "duration_sec": 30,
    "snaplen": 96,
    "filter": "tcp port 443"
  },
  "nonce": "<base64(random 16 bytes)>",
  "expires_at": "2025-09-24T12:05:00Z"
}

The Pivot Engine verifies the JWT, checks the nonce/replay cache (Redis with 5‑min TTL), and forwards the request to the Capture Orchestrator after appending an internal approval token (HMAC‑SHA256 keyed by a rotating secret).

Seamless Transition to Packet Capture

The Capture Orchestrator performs the following steps atomically (using a lightweight state machine persisted in etcd):

1. Reserve resources – check that the target device agent has < 2 active captures; if not, return 429 Too Many Requests.
2. Generate BPF filter – combine user‑provided filter with a mandatory not host <management-ip> to avoid capturing orchestration traffic.
3. Spawn tcpdump – via subprocess.Popen with stdout piped to a temporary file; set RLIMIT_FSIZE to enforce max size.
4. Stream metadata – every 5 s, upload a small JSON heartbeat ({ts, packets_captured, bytes_written}) to the object store; the UI polls this to show live progress.
5. Completion – on timeout or manual stop, finalize the pcap, compute SHA‑256, store the file under captures/<device>/<iface>/<alert_id>.pcap, and publish an event capture.completed to a Kafka topic.

The UI subscribes to this topic via a WebSocket bridge and displays a Download button and a Launch Analysis button that opens a tshark view in a modal.

Device Inspection and Analysis

After capture, the operator may request a device inspection. The workflow is:

1. Operator clicks Inspect Device → UI sends request to Pivot Engine with inspect:true.
2. Pivot Engine forwards to Device Agent, which:
   • Establishes an NETCONF session (or SSH fallback).
   • Executes a predefined read‑only command set:

<rpc message-id="101" xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <get>
    <filter type="subtree">
      <interfaces xmlns="urn:ietf:params:xml:ns:yang:ietf-interfaces">
        <interface>
          <name>GigabitEthernet0/1/0</name>
        </interface>
      </interfaces>
    </filter>
  </get>
</rpc>

• Optionally runs show logging | include GigabitEthernet0/1/0 via SSH.
• Returns structured JSON/XML to the UI, which renders tables and highlights anomalies (e.g., CRC errors, duplex mismatches).

All inspection commands are read‑only; the Device Agent rejects any request containing config or write keywords. If the operator later decides to apply a change (e.g., shut the interface), they must click Propose Change, which opens a change‑request form that requires a second‑factor approval and automatically generates a rollback plan (store current config, apply new config, schedule validation).

Automation and Trust Boundaries

Defining Trust Boundaries and Automation Limits

• Data‑plane boundary – Automation may only read telemetry (metrics, logs, packets). No write access to forwarding tables, ACLs, or QoS policies.
• Control‑plane boundary – Automation may issue show‑type commands and retrieve operational state. Configuration changes are outside the boundary and must be routed through an explicit approval workflow.
• Temporal boundary – Each capture session is bounded by a configurable max duration (default 30 s) and a hard stop after which the process is SIGTERM‑ed.
• Spatial boundary – Capture filters are constrained to exclude management IPs and to limit snaplen (default 96 bytes) to reduce payload exposure.

These limits are enforced at the Device Agent via capability dropping and seccomp profiles (e.g., disallow ptrace, mount).

Implementing Rollback Points and Safeguards

When an operator approves a configuration change, the workbench:

1. Pre‑change snapshot – via NETCONF get-config of running store, saved as <device>/<timestamp>/pre-change.cfg in an encrypted S3 bucket (SSE‑KMS).
2. Change execution – the approved config payload is pushed via NETCONF edit-config with default-operation="none"; the agent validates the candidate schema before applying.
3. Validation window – a configurable timer (default 300 s) starts; during this window the workbench continuously polls health metrics (interface error rate, packet loss). If any metric exceeds a threshold, an auto‑rollback is triggered.
4. Rollback execution – the pre‑change snapshot is pushed back with edit-config and the system logs a rollback event.
5. Manual abort – operator can click Abort & Rollback at any time, which immediately initiates step 4.

All snapshots are retained for a configurable retention period (default 14 days) and are subject to the same access‑control policies as capture files.

Ensuring Operator Oversight and Control

• Just‑in‑time (JIT) approval – each automation action (capture start, inspect, config change) requires a signed request that includes an operator‑generated nonce and short expiry (< 5 min). This prevents replay attacks and ensures a human is present at the moment of initiation.
• Session logging – every action writes an immutable log entry to an append‑only store (e.g., WORM‑enabled S3 or a write‑ahead log in Elasticsearch) containing operator identity, timestamp, action type, and justification.
• Role‑based access control (RBAC) – OIDC groups map to fine‑grained permissions (capture:init, inspect:device, change:approve).
• Approval workflow – for any config change, the UI presents a change‑request form that requires a second‑factor (e.g., push notification or hardware token) before the request is forwarded to the Pivot Engine.

These mechanisms collectively ensure that automation assists the operator without overstepping trust boundaries or removing the ability to rollback to a known‑good state.