An operator workbench for recursive path truth

NetOps Workbench Design

A sober, safety‑first workbench that correlates RIB routes, recursion graphs, and FIB install status before any automated change touches the network.

Core Requirements

• Visibility – real‑time, unified view of RIB, recursion graphs, and FIB state across all devices.
• Determinism – identical input yields identical correlation; no nondeterministic heuristics in safety‑critical paths.
• Blast‑radius control – remediation scoped to a well‑defined set of devices/prefixes with an explicit rollback path.
• Approval gating – automated fixes require explicit human‑in‑the‑loop approval (or policy‑based auto‑approve for low‑risk, pre‑validated actions).
• Auditability – immutable logs of every correlation, decision, and action with timestamps and operator identity.
• Extensibility – ingest data from NETCONF/gNMI, SNMP, syslog, BGP monitoring without proprietary agents.

Logical Blocks

Interfaces between blocks are versioned (protobuf over gRPC or JSON over HTTP). LLMs may appear only as an advisory UI layer; they never generate configuration directly.

Correlating Routes and Recursion Graphs

Route Correlation

For each destination prefix P, collect all candidate paths from OSPF, IS‑IS, BGP, static, etc. Each path record includes:

• Protocol & process ID
• Administrative distance / metric
• Next‑hop IP (or interface)
• Path attributes (AS‑path, communities, tag)
• Install state (candidate vs. selected)

Group by prefix and apply a deterministic best‑path algorithm that mirrors the device’s own rule (longest match → lowest admin distance → lowest metric → tie‑breaker). Implemented as a pure function; unit tests validate against show ip route on lab devices.

Recursion Graph Analysis

A recursion graph maps a prefix’s next‑hop dependencies until reaching a directly connected interface or a drop.

Construction

1. Extract the selected next‑hop NH for each prefix P.
2. Look up the RIB entry for NH (treated as a host prefix /32 or /128).
3. Repeat until the next‑hop resolves to a directly connected interface (via ARP/ND or interface address) or a null0/discard.
4. Record each hop as a node; edges represent “depends‑on”. Edge weight = hop metric (IGP cost, BGP MED, etc.).
5. Detect cycles with depth‑first search; any cycle triggers a recursion‑loop alert.

Stored in a graph database for fast traversal and UI visualisation. For scale, keep a prefix‑level summary (resolved outgoing interface, total path cost); materialise the full graph on demand.

Integrating Route and Recursion Data

In the Correlation Engine, for each prefix P:

1. Retrieve the selected RIB path and its next‑hop NH.
2. Query the recursion graph store for resolved outgoing interface IF and total cost C.
3. Compare IF with the interface implied by the FIB entry for P (see FIB section). A mismatch = FIB‑RIB inconsistency.
4. Emit a correlation record, e.g.:

{
  "prefix": "10.0.0.0/24",
  "selected_protocol": "BGP",
  "selected_metric": 100,
  "next_hop": "198.51.100.5",
  "resolved_if": "eth3",
  "resolved_cost": 100,
  "fib_installed": true,
  "fib_if": "eth3",
  "recursion_loop": false
}

Records are written to a time‑series table keyed by timestamp and device, enabling trend analysis and alerts on sudden increases in unresolved next‑hops.

FIB Install Status Monitoring

Overview

The FIB is the kernel/ASIC structure used for forwarding. A route is installed when the kernel/ASIC has programmed the entry; otherwise it resides only in the RIB. Discrepancies stem from:

• TCAM/FIB resource exhaustion
• RIB→FIB sync bugs
• Manual overrides bypassing the routing daemon
• Asynchronous updates during high churn

Monitoring Mechanisms

1. Push‑based telemetry – devices stream FIB entries via gNMI Subscribe or NETCONF notification. Example gNMI path:

   openconfig-routing:routing-instance/ribs/rib/fib
   openconfig-routing-policy:routing-policy/defined-sets/prefix-sets/prefix-set

   Collector normalises each entry to {prefix, next-hop, interface, metric, source-protocol}.

2. Pull‑based polling – lightweight script runs show ip fib (platform‑specific) via NETCONF get or CLI over SSH every 5‑15 s, depending on device capability and CPU budget.

Both streams feed the normalization layer; push is preferred for low latency, pull provides a safety net.

Alerting & Notification

An alert fires for prefix P on device D when any of the following hold:

• Missing FIB entry while RIB marks the route as selected.
• Mismatched next‑hop/interface between RIB and FIB.
• FIB entry present but marked inactive (hardware‑drop, TCAM miss).
• Sudden rise in FIB‑miss rate (> X % of total prefixes over a rolling window).

Alerts are emitted as Prometheus metrics (fib_missing_total, fib_mismatch_total) and forwarded to Alertmanager. Notification channels: email, Slack, ticketing webhook (creates a read‑only Jira issue with the correlation record attached). Payload includes:

• Device hostname
• Prefix
• RIB next‑hop & metric
• FIB state (missing/mismatched)
• Timestamp
• Link to the recursion‑graph snapshot for that prefix

Remediation Options and Automated Fixing

Manual Remediation Options

When an inconsistency is detected, the UI presents documented actions with impact and rollback steps.

Each action includes a pre‑filled change request template: exact command, expected state change, verification step (e.g., show ip route 10.0.0.0/24).

Automated Remediation Options

Automated fixes are limited to low‑risk, idempotent operations validated in a lab. The workbench proposes an automated fix only after:

1. Inconsistency persists for a configurable stabilisation window (default 2 min) to avoid reacting to transient flaps.
2. Recursion graph shows no loops and the next‑hop resolves to a directly connected interface or known stable transit.
3. Blast‑radius limits: ≤ N prefixes (default = 100) and ≤ M devices (default = 5).
4. A rollback snapshot exists and can be applied within R seconds (default = 30 s).

Allowed automated actions

• Static route injection for a missing FIB entry when the next‑hop is directly connected and the interface is up.
• Prefix‑list permit addition to restore a withdrawn advertisement caused by a mis‑configured filter.
• Metric adjustment (increase/decrease by a fixed value) to influence best‑path selection without altering policy.
• BGP community rewrite to influence upstream path selection (only if the device honors the community).

All actions are generated by a deterministic planner that outputs a vendor‑neutral change set (e.g., OpenConfig JSON). An adapter layer translates the set to device‑specific CLI or gNMI set payloads.

Approval Process for Automated Fixes

Before any automated change reaches the network, a two‑stage gate is enforced:

1. Policy‑based pre‑check – an automated policy engine evaluates the change set against rules (e.g., “never modify routes with community 65000:100”, “never install a static route pointing to RFC1918 on an external‑facing interface”). Rejection blocks the change and raises an alert.

2. Human‑in‑the‑loop (HITL) gate – if the policy passes, the UI shows:

   • Diff of intended configuration.
   • Estimated impact (prefix count, device count, expected traffic shift).
   • Rollback button that reverts to the stored snapshot.
   • Comment field for operator justification or modification request.

   The operator clicks Approve or Reject. Approval triggers the Execution Engine; rejection logs the reason and may create a follow‑up ticket.

All approval actions are immutably logged (user, timestamp, change‑set hash, decision) to satisfy audit requirements.

Troubleshooting and Debugging

Use the workbench UI to pull the relevant correlation record, recursion‑graph snapshot, and FIB state for the affected prefix and device, then follow the steps above to isolate and remediate the issue.