Score diagnosis process, not only final answer

Introduction to Network Copilot Evaluation

Accurate fact gathering is the foundation of reliable network troubleshooting or design validation. When a copilot asserts a fact—e.g., “interface GigabitEthernet0/0/0 is administratively down” or “BGP peer 10.0.0.2 is in Established state”—subsequent hypothesis testing and remedial actions depend on its veracity. Inaccurate or incomplete facts propagate errors downstream, leading to wasted effort, misguided configuration changes, and service‑impacting mistakes. Evaluation must therefore measure how completely and correctly the copilot extracts observable state from the control plane and management interfaces.

Beyond fact collection, a copilot must formulate plausible hypotheses that explain observed symptoms and systematically eliminate those inconsistent with the evidence. Effective hypothesis testing follows a falsification‑oriented approach: each hypothesis remains provisional until disproved by a concrete observation. Quality is judged by how rigorously the copilot rules out alternatives using logical inference and verifiable data, not by whether it eventually arrives at the correct answer. Rewarding a lucky final answer without examining the elimination path encourages over‑fitting to superficial patterns and undermines trust in the copilot’s reasoning.

Grading Criteria for Network Copilot

Design Trade‑offs

• Binary pass/fail – Simple but provides no granularity; a copilot that gathers 80 % of relevant facts scores the same as one that gathers 20 %.
• Weighted scoring model – Assigns numeric weights to fact completeness, hypothesis coverage, and justification quality, enabling differentiated feedback and trend analysis.
• Bayesian belief update – Treats each fact as evidence updating a posterior probability over hypotheses; theoretically sound but requires priors and likelihood models that are difficult to calibrate across heterogeneous vendor equipment.

Chosen direction – Weighted scoring model, because it balances interpretability with sensitivity to partial correctness, aligns with the evidence contract (observable facts → scores), and can be implemented with existing logging and telemetry infrastructure.

Rejected alternatives – Binary pass/fail was rejected for its inability to guide incremental improvement. The Bayesian approach was rejected due to the operational overhead of maintaining vendor‑specific likelihood tables and the risk of conflating model uncertainty with copilot performance.

Risks, observability, and rollback

• Risk: Mis‑calibrated weights could over‑emphasize easily measurable facts (e.g., interface status) while undervaluing deeper hypothesis work. Mitigation: Periodic weight review based on expert judgment and outcome correlation.
• Observability needs: Structured logs of (a) facts gathered, (b) hypotheses considered, (c) evidence used to accept/reject each hypothesis, and (d) the final justification for the next step. Export via syslog, gNMI, or custom JSON endpoints.
• Rollback: If weighting proves counterproductive, revert to a uniform weight (simple sum) or to the binary pass/fail baseline without changing data collection pipelines.

Fact Gathering and Verification

Methods for Verifying Facts

Fact verification cross‑checks the copilot’s asserted state against independent sources of truth:

1. Capture asserted fact – e.g., copilot logs “OSPF neighbor 10.1.2.3 is FULL”.
2. Retrieve ground‑truth – issue a vendor‑specific CLI or telemetry query that returns the same object.
3. Compare values – exact match for enumerated states, numeric tolerance for counters, semantic equivalence for complex structures (e.g., LSA headers).
4. Record outcome – binary pass/fail per fact, plus a confidence score if the source provides a reliability indicator (e.g., telemetry sampling rate).

Verification must be idempotent and non‑intrusive; read‑only commands are preferred to avoid perturbing the network state during evaluation.

Tools and Techniques for Fact Gathering

A copilot may draw facts from several planes:

• Control plane – routing protocol adjacency tables, LSAs, BGP updates.
• Management plane – device configuration, interface counters, SNMP/NETCONF/YANG datastores.
• Data plane – packet counters, NetFlow/IPFIX, telemetry streams (gNMI, Prometheus).

Common CLI‑based fact‑gathering techniques (adjust for vendor OS):

# Show OSPF neighbor state (Cisco IOS-XR)
show ospf neighbor

# Show BGP summary (Juniper Junos)
show bgp summary

# Verify interface admin/oper status (Linux‑based NOS)
ip link show dev eth0

# Retrieve YANG container for OSPF interfaces (gNMI Get)
gnmic -a <target>:9339 --username admin --password <pwd> \
      get --path "/interfaces/interface[name=eth0]/ospf"

Automated fact collection can be orchestrated via frameworks such as Ansible, Napalm, or custom Python scripts using Netmiko/NAPALM. The evaluation framework ingests the copilot’s fact log and compares each entry to the output of the corresponding verification command, producing a per‑fact correctness metric.

Hypothesis Testing and Elimination

Strategies for Ruling Out Incorrect Hypotheses

A robust hypothesis‑testing strategy follows these steps:

1. Enumerate candidate hypotheses – based on symptom patterns, recent changes, and known failure modes.
2. Derive observable predictions – each hypothesis predicts a set of facts that should be true if the hypothesis holds (e.g., “If OSPF area mismatch, then hello packets will contain different Area IDs”).
3. Gather evidence – issue targeted verification commands to confirm or refute each prediction.
4. Apply logical elimination – discard any hypothesis whose prediction is contradicted by verified facts.
5. Iterate – remaining hypotheses may be refined or combined; repeat until a single hypothesis survives or the set is exhausted.

Evaluation scores the copilot on:

• Hypothesis coverage – proportion of plausible hypotheses considered relative to an expert‑derived baseline.
• Elimination rigor – fraction of incorrect hypotheses correctly ruled out using concrete evidence.
• Evidence relevance – whether the facts used to eliminate a hypothesis are directly predictive of that hypothesis (avoiding circular reasoning).

Common Pitfalls in Hypothesis Testing

• Confirmation bias – favoring facts that support a preferred hypothesis while ignoring disconfirming evidence.
• Over‑reliance on correlation – treating temporal coincidence (e.g., a config change and a symptom) as causation without mechanistic validation.
• Incomplete hypothesis space – omitting rare but plausible causes (e.g., hardware‑level CRC errors) leading to premature convergence on an incorrect root cause.
• Misinterpreting transient states – treating a flapping adjacency as stable, causing false eliminations.

Evaluation logs should flag instances where the copilot retained a hypothesis despite a contradictory fact, or where it eliminated a hypothesis using unrelated evidence.

Evaluating Justification for Next Steps

Analyzing the Decision‑Making Process

After hypothesis elimination, the copilot must propose a concrete next action (e.g., “apply OSPF area‑ID correction on router R3”, “increase BGP hold‑time on peer 10.0.0.5”). The justification should explicitly map:

• Remaining hypothesis → required remediation → expected state change.

Each link in the chain must be backed by a verified fact or a protocol invariant (e.g., “OSPF will not form adjacency if Area IDs differ; correcting the Area ID on R3 restores matching IDs, which per OSPF RFC 2328 permits FULL state”).

Evaluation scores justification on:

• Logical completeness – all steps from hypothesis to action are explicitly stated.
• Evidence anchoring – each step cites a specific fact gathered earlier or a well‑known protocol rule.
• Actionability – the proposed step is executable with available privileges and tools.

Avoiding Intuition‑Based Decision Making

Intuition‑driven steps (e.g., “let’s just reboot the router”) lack traceable evidence and often mask gaps in hypothesis testing. The evaluation penalizes any justification that relies on unverified assumptions, vague statements (“it seems like…”), or actions not derivable from the surviving hypothesis. Metrics include the proportion of justification sentences that contain a direct fact citation or protocol reference.

Comparing Planned vs. Actual Outcomes

Once the copilot’s recommended action is executed (in a lab or change‑controlled window), the actual outcome is compared to the predicted outcome:

• Planned outcome – the state change the copilot asserted would result from its action (e.g., “OSPF neighbor will transition to FULL”).
• Actual outcome – the observed state after execution, gathered via the same verification commands used for fact collection.

Discrepancies are classified as:

• False positive – action predicted improvement but no change or degradation occurred.
• False negative – action predicted no effect but a beneficial change was observed (indicating a missed hypothesis).
• Correct – observed change matches prediction.

The evaluation logs the discrepancy type, performs root‑cause analysis (e.g., missing hypothesis about a downstream ACL), and updates weighting or hypothesis generation rules accordingly. This feedback loop transforms the copilot from a static responder into a learning system.

Troubleshooting Common Issues

Inaccurate or Incomplete Fact Gathering

• Missing interface state – copilot reports “link up” while show interfaces shows administratively down.
• Stale BGP routes – copilot assumes a route is active when the routing table shows it as withdrawn or suppressed.

Address these gaps by ensuring the copilot pulls the latest state from authoritative sources and timestamps each fact to detect staleness.

End of document.