Pre-merge verification gates for copilot changes

Introduction to Staged Release Pattern

Network Copilot is an AI‑assisted operational layer that consumes three types of artifacts: natural‑language prompts, large‑language model (LLM) weights, and tool‑adapter code that maps copilot intent to device‑specific CLI or API calls. Changes to any of these artifacts alter the copilot’s behavior when it interacts with live network equipment. Because the copilot can issue configuration‑changing commands, a faulty update can cause service‑impacting misconfigurations, security policy violations, or device lock‑outs. Therefore, the release process must treat each artifact change as a controlled transaction with explicit pre‑conditions, verification gates, and rollback criteria.

Benefits of a Staged Release Pattern

• Blast‑radius containment – updates are first applied to a limited set of devices or synthetic test beds, limiting the impact of a defect.
• Verification‑driven progression – each stage requires objective evidence (test results, telemetry thresholds) before the change can advance.
• Operator‑in‑the‑loop approval – manual intervention points ensure that human expertise validates ambiguous or high‑risk changes.
• Clear rollback semantics – failure at any gate triggers a well‑defined rollback path that restores the previous known‑good artifact set.
• Auditability – each gate logs inputs, outputs, and decisions, supporting post‑mortem analysis and compliance reporting.

Designing the Staged Release Pattern

Architecture of the Staged Release System

The staged release system consists of four logical components that enforce transaction boundaries:

1. Ingress Queue – receives candidate updates (prompt, model, adapter) from CI pipelines. Each entry carries a unique change ID, artifact hashes, and a reference to the engineering case it intends to satisfy.
2. Validation Engine – runs deterministic pre‑checks (schema validation, hash integrity) and executes engineering‑case‑specific test suites. This component defines the pre‑check boundary.
3. Gatekeeper Service – evaluates validation results against configurable thresholds. If all gates pass, it issues a commit signal that promotes the update to the next stage; otherwise it triggers a rollback signal.
4. Rollout Controller – stages the promoted update across environments (synthetic lab → canary devices → production fleet) using a pacing algorithm. After each stage it invokes a verification gate (telemetry, health checks) before allowing progression.

Each component interacts via immutable message objects, ensuring that the transaction scope is limited to the set of artifacts bound to a single change ID. No cross‑talk between concurrent changes is permitted unless explicitly batched by the operator.

Engineering Case Validation

Engineering cases are concrete, reproducible scenarios that capture known failure modes, feature requirements, or regulatory constraints (e.g., “prompt must not generate a no shutdown command on interfaces in shutdown‑only mode”, “model must answer BGP‑peer‑state queries with <95% accuracy on a validation set”, “adapter must translate show ip route to the correct vendor‑specific CLI”). Validation proceeds as follows:

• Case Retrieval – the Validation Engine pulls the case definition from a version‑controlled repository (Git) tagged with the change’s target release.
• Test Execution – automated test runners (e.g., pytest, Robot Framework) execute the case against a sandbox that mirrors the target device OS version. The sandbox may be a virtual router, a containerized CLI emulator, or a hardware‑in‑the‑loop test bed.
• Result Aggregation – each case yields a PASS/FAIL status and a set of metrics (latency, command correctness, side‑effect flags). The Validation Engine records the raw output and computes a pass‑rate.
• Gate Decision – the Gatekeeper compares the pass‑rate to a pre‑defined threshold (commonly 100% for safety‑critical cases, ≥95% for performance‑related cases). Failure to meet the threshold aborts the transaction and surfaces the failing case IDs to the operator.

Because engineering cases are versioned and immutable at the time of validation, the pre‑check boundary is repeatable: re‑running the same validation with the same inputs yields identical outcomes.

Gatekeeping Mechanisms for Updates

Gatekeeping enforces the commit boundary and provides the operator intervention point. Mechanisms include:

• Automatic Gates – thresholds on test pass‑rates, artifact hash verification, and dependency compatibility (e.g., adapter version must declare support for the model’s API version). These are evaluated without human input.
• Manual Approval Gate – after automatic gates succeed, a change ID is placed in a review queue. An authorized network reliability engineer (NRE) must examine the engineering case results, artifact diffs, and a risk‑assessment checklist before issuing a manual “promote” command.
• Time‑Based Window – updates may only be promoted during predefined maintenance windows to limit blast‑radius exposure to off‑peak traffic.
• Rollback Trigger – if any verification gate (post‑promotion telemetry, health‑check) fails, the Gatekeeper automatically issues a rollback signal. The rollback is operational: it commands the Rollout Controller to redeploy the previously approved artifact set (prompt, model, adapter) to the affected stage. Automatic rollback of model weights is not guaranteed; if the new model cannot be safely reverted, the system flags a manual intervention requirement.

Implementing the Staged Release Pattern

Code Examples for Update Validation

Below is a simplified Python snippet that demonstrates the Validation Engine’s pre‑check and engineering‑case execution for a prompt update. It assumes the existence of a case_loader module that returns a list of test functions.

# validation_engine.py
import hashlib
import json
import os
from typing import List, Callable
from case_loader import load_cases  # loads cases from Git tag

ARTIFACT_DIR = "/var/copilot/artifacts"
CHANGE_ID = os.getenv("CHANGE_ID")  # injected by CI


def verify_hash(artifact_path: str, expected_hash: str) -> bool:
    sha256 = hashlib.sha256()
    with open(artifact_path, "rb") as f:
        for chunk in iter(lambda: f.read(4096), b""):
            sha256.update(chunk)
    return sha256.hexdigest() == expected_hash


def run_engineering_cases(case_dir: str) -> List[dict]:
    cases: List[Callable[[], dict]] = load_cases(case_dir)
    results = []
    for case_fn in cases:
        try:
            result = case_fn()  # returns {"pass": bool, "metrics": {...}}
            results.append(result)
        except Exception as exc:
            results.append({"pass": False, "error": str(exc)})
    return results


def main(change_id: str):
    # Load manifest describing the change
    with open(f"{ARTIFACT_DIR}/{change_id}/manifest.json") as f:
        manifest = json.load(f)

    # Pre‑check: hash integrity
    for artifact, info in manifest["artifacts"].items():
        path = f"{ARTIFACT_DIR}/{change_id}/{artifact}"
        if not verify_hash(path, info["sha256"]):
            raise ValueError(f"Hash mismatch for {artifact}")

    # Engineering case validation
    case_results = run_engineering_cases(f"{ARTIFACT_DIR}/{change_id}/cases")
    pass_rate = sum(1 for r in case_results if r.get("pass")) / len(case_results)

    # Emit validation report for Gatekeeper
    report = {
        "change_id": change_id,
        "pass_rate": pass_rate,
        "case_results": case_results,
    }
    print(json.dumps(report))  # consumed by Gatekeeper via stdout


if __name__ == "__main__":
    main(CHANGE_ID)

Key points:

• The hash verification constitutes the pre‑check boundary; failure aborts before any test runs.
• Engineering case execution is isolated; each case returns a deterministic pass/fail.
• The validation report is the artifact that the Gatekeeper consumes to decide whether to cross the commit boundary.

CLI Commands for Manual Intervention

Operators interact with the staged release system via the copilotctl CLI. Below are the essential commands for manual gates and rollback.

# 1. Prompt a change to await manual approval after automatic gates pass
copilotctl review start --change-id <CHANGE_ID> --reason "Awaiting NRE sign‑off"

# 2. List pending reviews
copilotctl review list --state pending

# 3. Approve a change (manual gate)
copilotctl review approve --change-id <CHANGE_ID> --operator $(whoami) --comment "All safety cases passed"

# 4. Reject a change (triggers rollback to previous known‑good version)
copilotctl review reject --change-id <CHANGE_ID> --operator $(whoami) --comment "Prompt generated unsafe CLI"

# 5. Force a rollback of a staged rollout (operational, not automatic)
copilotctl rollout rollback --change-id <CHANGE_ID> --stage canary --to-previous

# 6. View the current artifact versions deployed per stage
copilotctl rollout status --change-id <CHANGE_ID>

Each command writes an audit entry to /var/log/copilot/audit.log with timestamps, operator identity, and the exact state transition, satisfying the requirement for explicit operator intervention points.

Automated Testing for Engineering Cases

The CI pipeline invokes the validation script as a job. A representative GitHub Actions workflow snippet:

name: Validate Copilot Update

on:
  push:
    branches: [main]
    paths:
      - 'artifacts/**'
      - 'cases/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt
      - name: Run validation engine
        env:
          CHANGE_ID: ${{ github.sha }}
        run: |
          python validation_engine.py ${{ github.sha }} > validation_report.json
      - name: Upload report as artifact
        uses: actions/upload-artifact@v3
        with:
          name: validation-report-${{ github.sha }}
          path: validation_report.json

The workflow fails if the validation script exits with a non‑zero status, ensuring that the pre‑check boundary is enforced before any artifact is promoted to the gatekeeper service.

Troubleshooting the Staged Release Pattern

Common Issues and Error Messages

Debugging Techniques for Update Failures

1. Inspect Validation Logs – The validation engine writes detailed stdout/stderr to the CI artifact; retrieve it and grep for the failing case name.
2. Reproduce the Case Locally – Clone the engineering case repository at the exact Git tag used in the manifest and run pytest -k <case_name> -s to see raw output.
3. Check Artifact Propagation – Verify that the artifact hashes in the manifest match those stored in the artifact repository (sha256sum <file>).
4. Telemetry Deep‑Dive – If verification gate fails, pull the time‑series metrics from the monitoring system (Prometheus) for the canary devices and correlate with the exact timestamp of the update push.
5. Gatekeeper Decision Trace – Enable debug logging on the Gatekeeper service (LOG_LEVEL=debug) to see which gate threshold caused rejection.

Rollback Procedures for Failed Updates

Rollback is operational, not automatic, for model and prompt changes because reverting weights may require re‑loading a previous model file and restarting the copilot service. The procedure:

1. Detect Failure – Verification gate (health check, telemetry, or manual observation) returns a non‑OK status.
2. Trigger Rollback Signal – The Gatekeeper publishes a ROLLBACK message on the internal message bus, containing the change_id and the target stage.
3. Rollout Controller Response – Upon receipt, the controller:
   • Stops further promotion of the change.
   • Issues a copilotctl rollout rollback --change-id <ID> --stage <STAGE> --to-previous command.
   • The controller redeploys the previous artifact set (identified by the previous_change_id stored in the rollout metadata) to the affected stage.
   • Restarts the copilot service on the target devices using the orchestration platform (e.g., Kubernetes rollout or Ansible playbook).
4. Verification After Rollback – The same verification gate is re‑run; success is required before the controller considers the stage healthy.
5. Operator Notification – An alert is sent to the on‑call NRE with details of the failure, the rollback action taken.