Rollback plans need their own gates

Introduction

AI‑generated rollback is a machine‑learned or rule‑based artifact that encodes the inverse of a planned configuration change. Produced automatically from the change request, the current device state, and a model of system behavior, it provides a repeatable, version‑controlled mechanism for restoring the pre‑change state when a change fails verification gates or when an operator aborts within a defined change window.

Treating this rollback as a first‑class artifact brings versionability, expiry awareness, reentry governance, operational clarity, and risk‑based gating to change management.

Benefits of Treating AI‑Generated Rollback as a First‑Class Artifact

• Versionability – Stored alongside the change artifact in Git or an artifact registry, enabling traceability from change → rollback → verification.
• Expiry Awareness – Explicit expiry metadata guarantees the rollback is not used after the captured prerequisite state becomes invalid.
• Reentry Governance – Encoded reentry rules prevent accidental re‑application after the system has moved forward, reducing rollback‑loop risk.
• Operational Clarity – The rollback lifecycle (capture, validation, storage, invocation) is visible in CI/CD pipelines and change‑management tools, supporting audit and compliance.
• Risk‑Based Gating – Automated validation (diff, simulation) before the change window opens provides an early‑stop signal if the rollback cannot be trusted.

Prerequisite Capture for AI‑Generated Rollback

Identifying Critical System Components

1. Configuration Sources – NETCONF/YANG models, CLI snippets, Ansible playbooks, Terraform state, device‑local configs.
2. Runtime State – BGP neighbor states, interface counters, QoS statistics, ARP/ND tables, etc.
3. Dependencies – RADIUS/TACACS+ servers, NTP peers, DNS resolvers, and other external services whose state must remain unchanged.
4. Change Blast Radius – Set of devices, interfaces, and logical entities the change will touch; prerequisite capture must cover the full blast radius.

Capturing Prerequisite Data

• Snapshot Mechanism – Use a device‑agnostic tool (e.g., pyATS genie parse, Napalm get, or vendor‑specific show running-config | json) to produce structured JSON/YAML.
• Immutable Storage – Write each snapshot to a write‑once object store keyed by:

  <change-id>/<component-type>/<device-hostname>/<timestamp>.json

  Example using AWS S3 with Object Lock and Glacier IR storage class.
• Metadata Enrichment – Attach to each artifact:
  • capture_time (ISO‑8601 UTC)
  • change_id (UUID from ITSM)
  • git_sha (commit that generated the change)
  • tool_version (version of the capture utility)
  • hash (SHA‑256 of snapshot content for integrity verification)
• Validation Step – After capture, run a lightweight sanity check (JSON parse, mandatory keys) and fail the pipeline on any failure.

Integrating Prerequisite Capture into CI/CD

# Example: GitHub Actions workflow snippet
name: Change & Rollback Preparation
on:
  workflow_dispatch:
    inputs:
      change_id:
        description: 'Change identifier from ITSM'
        required: true

jobs:
  capture-prereqs:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v4

      - name: Install capture tools
        run: |
          pip install pyats[genie] napalm

      - name: Capture device snapshots
        id: snapshots
        env:
          CHANGE_ID: ${{ github.event.inputs.change_id }}
        run: |
          mkdir -p artifacts/${CHANGE_ID}
          for dev in $(cat inventory.txt); do
            pyats learn --testbed testbed.yaml --device $dev \
              --output artifacts/${CHANGE_ID}/$dev.json
          done
          # Generate SHA‑256 manifests
          find artifacts/${CHANGE_ID} -type f -name '*.json' -exec sha256sum {} \; > artifacts/${CHANGE_ID}/manifest.sha256

      - name: Upload artifacts to immutable store
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      - run: |
          aws s3 sync artifacts/${CHANGE_ID} s3://rollback-artifacts/${CHANGE_ID}/ \
            --storage-class GLACIER_IR \
            --object-lock-mode GOVERNANCE \
            --object-lock-retain-until-date 2099-12-31T00:00:00Z

      - name: Validate capture integrity
        run: |
          aws s3 cp s3://rollback-artifacts/${CHANGE_ID}/manifest.sha256 - | sha256sum -c -

This workflow ensures prerequisite data is captured, cryptographically sealed, and stored before any change proceeds.

Expiry Checks for AI‑Generated Rollback

Time‑Based Expiry

• Expiry Timestamp – Store an expires_at field (UTC) in the artifact’s metadata. Typical values:
  • Hot‑fix: now + 2h
  • Standard maintenance window: now + 24h
  • Long‑term baseline: now + 30d (rare for AI‑generated rollback)
• Check Logic – Before invoking rollback, compare current UTC time (now_utc) with expires_at. If now_utc > expires_at, the rollback is stale and must be regenerated.

Version‑Based Expiry

• Change Version Tag – Derive a version string from the Git commit (e.g., v2024.09.26-03a7f1c). Record this as change_version in the rollback artifact.
• Expiry Condition – The rollback is valid only if the current repository state is at an equal or earlier commit. A newer commit that modifies the same configuration path invalidates the rollback.
• Implementation – After capturing prerequisites, record:

  {
    "change_version": "v2024.09.26-03a7f1c",
    "repo_url": "https://git.example.com/netops/configs.git",
    "repo_branch": "main"
  }

  Validation script:

  CURRENT_SHA=$(git ls-remote $REPO_URL $REPO_BRANCH | cut -f1)
  if [[ "$CURRENT_SHA" != "$RECORDED_SHA" ]]; then
    echo "Rollback expired: repo has advanced beyond recorded version"
    exit 1
  fi

Handling Expiry Check Failures

• Fail‑Fast – Abort the change workflow before the window opens if either time‑ or version‑based expiry fails. Log: reason (time_expired or version_mismatch), current timestamps/versions, and link to the stale artifact.
• Operator Override (Optional) – A break‑glass procedure may allow forced usage after dual approval and risk documentation; the override must be recorded immutably and trigger a post‑incident review.
• Regeneration Trigger – On expiry failure, automatically initiate a new prerequisite capture and AI‑generated rollback for the same change ID (if still pending) or create a new change request.

Reentry Rules for AI‑Generated Rollback

Defining Reentry Conditions

A rollback must not be reapplied after the system has progressed beyond its intended revert point. Conditions include:

1. State Divergence – Current device state differs from the captured prerequisite state outside the original change scope (e.g., unrelated configuration drift).
2. Subsequent Change Applied – A later change with a higher version/timestamp touches any part of the rollback’s scope.
3. Health Check Failure – Post‑change verification indicates an unhealthy state; rolling back could worsen the issue.
4. Manual Lock – Operator‑set reentry lock (feature flag or lock object) to prevent rollback reuse.

Implementing Reentry Rules

Encode conditions as a JSON policy evaluated by the rollback orchestrator:

{
  "reentry_rules": [
    { "type": "state_match", "scope": "prereq_snapshot", "operator": "equals" },
    { "type": "version_lt", "field": "change_version", "operator": "<", "value": "current_change_version" },
    { "type": "health_check", "script": "verify_health.sh", "expected_exit": 0 }
  ]
}

Evaluation Flow

1. Load the prerequisite snapshot from the artifact store.
2. Execute each rule; if any returns false, abort and raise a reentry‑violation event.
3. If all pass, proceed with rollback application.

Handling Reentry Rule Violations

• Immediate Abort – Halt rollback, raise an alert (PagerDuty, Slack) with: failed rule, current vs. expected values, artifact link, change request link.
• Manual Review Queue – Place the change in a “rollback‑blocked” state requiring CAB review before further action.
• Automatic Remediation (Optional) – For minor, harmless drift (e.g., a counter), optionally generate a delta rollback that reverts only the original change’s specific lines, leaving other drift untouched; the delta must be separately validated.
• Audit Logging – Record violation event with: timestamp, actor, rule ID and outcome, artifact hash, decision (abort, manual review, delta‑generated).

Integrating AI‑Generated Rollback into Change Management

Creating a Change Window

1. Define Window Boundaries – Agree on window_start and window_end in the change request. The window must accommodate:
   • Prerequisite capture (if not pre‑done)
   • AI‑generated rollback creation and validation
   • Change execution
   • Verification gates
   • Rollback execution window (typically 15‑30 min after change completion)
2. Lock the Window – Use the change‑management system (ServiceNow, Jira Service Management) to set the change to “Scheduled” and block overlapping windows on the same CI/device group.
3. Communicate Expiry – Publish the rollback’s expires_at on the change ticket; the window must close before this time.

Validating AI‑Generated Rollback Artifacts

• Integrity Check – Verify SHA‑256 hash against the manifest stored with the artifact.
• Schema Validation – Ensure conformance to a predefined JSON schema (e.g., using ajv or jsonschema):

  {
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "AIGeneratedRollback",
    "type": "object",
    "required": ["change_id", "prereq_snapshot", "metadata"],
    "properties": {
      "change_id": { "type": "string" },
      "prereq_snapshot": { "type": "object" },
      "metadata": {
        "type": "object",
        "required": ["capture_time", "expires_at", "change_version", "hash"],
        "properties": { ... }
      }
    }
  }

• Semantic Validation – Run a dry‑run simulation (e.g., pyATS or Ansible --check) to confirm that applying the rollback reproduces the prerequisite snapshot within an acceptable tolerance (ignoring transient counters).
• Approval Gate – Require sign‑off from the Change Manager and, for high‑risk changes, from a Network Reliability Engineer before marking the artifact “valid for apply”.

Applying AI‑Generated Rollback during Change Window

# Pseudocode for the orchestration step
ROLLBACK_ARTIFACT="s3://rollback-artifacts/${CHANGE_ID}/rollback.json"
# 1. Download and verify
aws s3 cp $ROLLBACK_ARTIFACT /tmp/rollback.json
sha256sum -c /tmp/rollback.manifest

# 2. Run reentry rule evaluation (python script)
python /opt/rollback/validate_reentry.py \
    --artifact /tmp/rollback.json \
    --inventory /etc/ansible/inventory.yml \
    --health-script /opt/rollback/verify_health.sh

if [ $? -ne 0 ]; then
    echo "Reentry rules failed – aborting rollback"
    exit 1
fi

# 3. Apply rollback (idempotent where possible)
ansible-playbook -i /etc/ansible/inventory.yml \
    /opt/rollback/playbooks/apply_rollback.yml \
    -e "artifact=/tmp/rollback.json"

# 4. Post‑apply verification
python /opt/rollback/verify_state.py --expected /tmp/rollback.json --actual /tmp/current_state.json

This sequence guarantees:

• Artifact integrity and non‑expiry.
• Satisfaction of reentry rules.
• Controlled, idempotent rollback application.
• Final verification that the system matches the prerequisite state (within allowed tolerances).

Troubleshooting AI‑Generated Rollback Issues

General Guidance

• Verify artifact location and access permissions.
• Re‑run the capture workflow if integrity or expiry checks fail.
• Consult the audit log for detailed failure context before invoking any override.
• For persistent issues, engage the platform team to review snapshot tool versions and storage lock policies.