LinkState
Go back

Running centralized and distributed gateways during migration

Introduction to Brownfield Fabrics

Definition and Characteristics

A brownfield fabric is an existing network infrastructure that has been operational for a period of time, often running legacy control planes, mixed hardware generations, and heterogeneous software releases. Characteristics include:

Challenges in Brownfield Environments

  1. Partial visibility – telemetry may be missing on older devices, making it hard to confirm state after a change.
  2. Inconsistent rollback capability – some platforms support atomic commit/rollback (e.g., Junos commit‑confirmed), others only allow manual reverse‑apply.
  3. Blast‑radius uncertainty – a change that touches a shared control plane (e.g., BGP route‑reflector) can affect tenants that are not yet migrated.
  4. First‑hop ownership ambiguity – when both legacy L2 gateways and new VXLAN VTEPs advertise the same subnet, hosts may ARP‑resolve to either device, causing blackholing or duplicate‑address detection.
  5. Change‑window pressure – business units often demand zero‑downtime migrations, yet the fabric cannot be taken offline for a flag‑day cutover.

These challenges drive the need for explicit transaction boundaries, failure‑domain isolation, and clear first‑hop ownership strategies when coexisting old and new fabrics.

Coexistence Patterns for Brownfield Fabrics

Parallel Deployment Model

The new fabric runs side‑by‑side with the legacy fabric for the same tenant workloads. Both fabrics forward traffic independently; traffic is steered via policy‑based routing (PBR) or service‑insertion points.

Key properties

Typical CLI (Cisco NX‑OS)

# Define a class-map for tenant 101 traffic
class-map match-any TENANT_101
  match destination-address 10.10.101.0/24
  match source-address 10.10.101.0/24
exit

# Policy-map that sends traffic to the new VXLAN VTEP (next-hop 192.168.200.1)
policy-map PBR_TENANT_101
  class TENANT_101
    set ip next-hop 192.168.200.1
exit

# Apply PBR on the legacy SVI interface
interface Vlan101
  service-policy input PBR_TENANT_101

Staggered Deployment Model

Migration proceeds tenant‑by‑tenant (or subnet‑by‑subnet) with a hard cut‑over: the legacy fabric stops forwarding for that tenant once the new fabric is verified. The old and new fabrics never forward the same tenant traffic simultaneously.

Key properties

Typical CLI (Arista EOS)

# Withdraw legacy SVI for tenant 202
interface Vlan202
  shutdown
exit

# Create VNI-backed VRF on new fabric
vrf definition TENANT_202
  rd 65000:202
  address-family ipv4 unicast
    route-target import 65000:202
    route-target export 65000:202
  exit-address-family
exit

vlan 202
  name TENANT_202
  vmware [vsphere]  # optional
exit

interface Vxlan1
  vxlan udp-port 4789
  vxlan vni 202 map VLAN 202
exit

Hybrid Deployment Model

Combines parallel and staggered approaches: a tenant is first run in parallel (dual‑forwarding) for validation, then the legacy path is gracefully drained while the new path takes over, and finally the legacy path is removed. This provides a safety net for applications sensitive to brief packet loss or re‑ordering.

Key properties

Typical CLI (Juniper Junos – using routing‑instance and firewall filter)

# Stage 1: Install a firewall filter that load‑balances 50/50 between old and new next‑hops
firewall {
    family inet {
        filter F_TENANT_303 {
            term BALANCE {
                from {
                    destination-address 10.10.303.0/24;
                    source-address 10.10.303.0/24;
                }
                then {
                    load-balance per-packet;
                }
            }
            term ACCEPT { then accept; }
        }
    }
}
apply-interfaces {
    ge-0/0/12.0 { family inet { filter { input F_TENANT_303; } } }
}

# Stage 2: Adjust weight (using routing‑instance preference)
routing-instances {
    TENANT_303 {
        instance-type vrf;
        interface ge-0/0/12.0;
        routing-options {
            autonomous-system 65000;
            forwarding-table {
                export [ TENANT_303_EXPORT ];
            }
        }
        protocols {
            bgp {
                group NEW_VTEP {
                    type external;
                    peer-as 65001;
                    neighbor 192.168.250.10 {
                        bfd-liveness-detection {
                            minimum-interval 300;
                        }
                    }
                    neighbor 192.168.250.20 {   # legacy gateway
                        bfd-liveness-detection {
                            minimum-interval 300;
                        }
                        # lower weight to prefer new path
                        local-preference 150;
                    }
                }
            }
        }
    }
}

Transaction Boundaries in Brownfield Fabrics

Understanding Transaction Boundaries

A transaction boundary defines the set of state changes that are treated as an atomic unit for the purpose of validation, commit, and rollback. In brownfield fabrics the boundary must respect:

If a change crosses a boundary (e.g., modifies a shared route‑reflector that serves both fabrics), the transaction must be expanded to include all affected tenants, or the change must be rejected.

Managing Transaction Boundaries in Coexistence Scenarios

  1. Define the boundary up‑front – document which VRFs/VNIs, which physical interfaces, and which control‑plane protocols are included.
  2. Encapsulate the change in an idempotent automation block – e.g., an Ansible playbook that first checks the current state, then applies only the delta.
  3. Insert explicit pre‑check and verification tasks – these act as the “commit gate”.
  4. Use platform‑native commit mechanisms where available (Junos commit confirmed, NX‑OS rollback with checkpoint, EOS atomic config replace).
  5. If native atomicity is absent, implement a manual two‑step: apply change → verify → if OK, issue a “commit” command; if not, run a pre‑saved rollback script.

Code Examples for Transaction Boundary Management

Ansible playbook (NX‑OS) – tenant cutover with explicit boundary

- name: Brownfield tenant cutover – transaction boundary
  hosts: nxos_switches
  gather_facts: false
  vars:
    tenant_vlan: 202
    tenant_vni: 202
    legacy_svi: "Vlan{{ tenant_vlan }}"
    new_vtep_ip: "192.168.200.10"
  tasks:
    - name: Pre‑check – verify legacy SVI is up
      nxos_command:
        commands:
          - "show interface {{ legacy_svi }} | include line protocol"
      register: svi_state
      failed_when: "'up' not in svi_state.stdout[0]"

    - name: Pre‑check – verify EVPN route for VNI exists on new fabric
      nxos_command:
        commands:
          - "show bgp l2vpn evpn | include {{ tenant_vni }}"
      register: evpn_state
      failed_when: evpn_state.stdout[0] | length == 0

    - name: Commit – move traffic to new VTEP via PBR
      nxos_config:
        lines:
          - "interface {{ legacy_svi }}"
          - "  service-policy input PBR_TENANT_{{ tenant_vlan }}"
        before: no service-policy input PBR_TENANT_{{ tenant_vlan }}
        after: service-policy input PBR_TENANT_{{ tenant_vlan }}
      register: commit_result

    - name: Verification gate – monitor loss for 300 seconds
      nxos_command:
        commands:
          - "show interface {{ legacy_svi }} counters | include packets"
      register: before_counters
    - pause:
        seconds: 300
    - name: Verification gate – compare counters
      nxos_command:
        commands:
          - "show interface {{ legacy_svi }} counters | include packets"
      register: after_counters
      # simple delta check – in real life use a threshold
      failed_when: >
        after_counters.stdout[0] | regex_search('in packets: (\d+)') |
        int < before_counters.stdout[0] | regex_search('in packets: (\d+)') |
        int

    - name: Rollback trigger – if verification fails, revert PBR
      nxos_config:
        lines:
          - "interface {{ legacy_svi }}"
          - "  no service-policy input PBR_TENANT_{{ tenant_vlan }}"
      when: verification_failed|default(false)

Failure Domains and Rollback Strategies

Identifying Failure Domains in Brownfield Fabrics

A failure domain is the set of devices or state that can be impacted by a single fault. In brownfield fabrics typical domains include:

When designing a transaction, map each changed object to its failure domain and ensure the domain is contained within the transaction boundary or that you have a compensating action for each domain.

Rollback Strategies for Coexistence Scenarios

  1. Checkpoint‑based rollback – platforms that support configuration checkpoints (Junos, NX‑OS) allow you to save a checkpoint pre‑change and roll back to it atomically.
  2. Reverse‑playbook – store the inverse of each automation task (e.g., no form of a command) and execute it on failure.
  3. Manual re‑apply – for platforms lacking native rollback (some older IOS‑XE), keep a copy of the pre‑change config and re‑apply via CLI or API.
  4. State‑based rollback – instead of config rollback, re‑apply the last known‑good state (e.g., MAC/IP bindings
Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email
Previous Post
Modeling MLAG, chassis, and virtual pairs honestly
Next Post
The remote SSH lockout after an atomic replace