Tenant Leaks in Partial Rollback Scenarios

Introduction

A tenant leak occurs when traffic that should be confined to a specific logical tenant (VRF, VPN, or security zone) is inadvertently allowed to traverse into another tenant’s forwarding domain. In Ethernet‑VPN (EVPN) fabrics with MP‑BGP control plane, isolation is enforced primarily by route‑targets (RTs) and filtering constraints (import/export policies or route‑maps).

A leak is partial when the misconfiguration is isolated to a subset of devices (e.g., a single leaf switch) while the rest of the fabric remains correctly programmed. The control plane continues to converge, so the underlay and overlay stay operational, but the data‑plane forwarding decision on the affected leaf deviates from the intended policy, allowing packets to cross tenant boundaries.

Common Causes

Understanding Route‑Targets and Filtering Constraints

Route‑Target Fundamentals

In EVPN, a route‑target is an extended community attached to BGP NLRI that signals which VRF should import or export the route.

• Export RT: When a PE (or leaf) originates a route for a local prefix, it attaches one or more export RTs.
• Import RT: A VRF is configured with a list of import RTs; any BGP route carrying a matching RT is imported into that VRF’s routing table.

Mathematically:

if (route.RT ∩ VRF.import_RTs ≠ ∅) then install route in VRF

RTs travel as an optional transitive BGP attribute; they do not affect underlay forwarding. The data‑plane enforcement occurs when the forwarding ASIC first derives the VRF context from the incoming interface/VLAN, then performs a longest‑prefix match (LPM) within that VRF’s FIB. If a route was incorrectly imported due to a missing or wrong RT, the packet is forwarded using that VRF’s FIB, potentially reaching a destination belonging to another tenant.

Filtering Constraint Basics

Filtering constraints are policy mechanisms applied at the BGP import/export stage to further restrict which routes are accepted based on criteria beyond RT matching (e.g., IP prefix, MAC address, VNI, or route‑type). Common implementations:

• Route‑maps with match ip address prefix-list / match mac-list statements.
• Import/export policies defined in the VRF configuration (import map, export map).
• EVPN‑specific constraints such as vni-only or encap-type checks.

A filtering constraint can be whitelisting (only allow matching routes) or blacklisting (deny matching routes). Evaluation order is deterministic:

1. RT match (mandatory for EVPN routes).
2. If RT matches, evaluate import/export route‑map (if present).
3. If the route‑map returns permit, the route is installed; otherwise, it is dropped.

If a leaf lacks an import map that should deny a certain prefix, the RT check alone may permit the route, leading to a leak.

Interaction Between Route‑Targets and Filtering Constraints

Install(route) = RT_match(route) AND (Filter(route) == permit OR no Filter configured)

• Both present: filter refines the RT‑based import.
• Only RT present: any route with a matching RT is imported unconditionally.
• Only filter present (no RT): filter is ineffective for EVPN because RT match is a prerequisite; the route is dropped before filter evaluation.

Thus, a missing RT is a hard failure (route never matches any VRF), whereas a missing or overly permissive filter is a soft failure (route matches RT but should have been blocked). In a partial rollback scenario, the leaf may have correct RTs but lack the intended filter, or vice‑versa, creating a leak that does not break overall BGP convergence but violates segmentation.

Identifying Partial Rollback Scenarios

Symptoms

Detection Methods

1. Policy‑as‑code diff

   • Render intended config (e.g., from Ansible/Terraform) for each leaf.
   • Compare with show running-config using batdiff or ntc‑templates.
   • Look for missing route-target import/export lines or missing import map statements.

2. BGP RT verification

   • On each leaf: show bgp vpnv4 vrf <VRF> | include <RT>
   • Zero matches for an expected RT indicates a missing import.

3. Forwarding plane validation

   • Use ping with specific source/destination and vary source port to force ECMP hash (e.g., ping -s 1472 -p 0xabcd).
   • Capture packets on both leaves with tcpdump -i <if> -w leak.pcap and verify VLAN/VNI tags.

4. Telemetry correlation

   • Pull sFlow/NetFlow records and group by src_vrf, dst_vrf.
   • Any non‑zero flow where src_vrf != dst_vrf and the flow is not in the approved communication matrix indicates a leak.

5. Automated drift detection

   • Tools like Network Source of Truth (NSoT), NetBox, or Slurp’it store intended RT/filter objects.
   • Periodic jobs run ansible‑network‑automation modules to compare intended vs. actual and raise alerts on divergence.

Tools and Techniques

Troubleshooting a Partial Rollback

Step‑by‑Step Guide

1. Define the observed symptom

   • Record exact source/destination IPs, VNI/VLAN, and time window where leakage occurs.
   • Example: 10.1.2.5 (VRF RED) → 10.2.3.8 (VRF BLUE) observed at 14:03–14:07 UTC.

2. Collect intended policy

   • Extract RT and filter definitions from the source‑of‑truth (e.g., Git repo).
   • Example intended snippet for leaf:

     vrf RED  
       address-family ipv4 unicast  
         import route-target 65000:100  
         import map RED-IMPORT  
       !  
     route-map RED-IMPORT permit 10  
       match ip address prefix-list RED-ALLOWED  
     !  
     ip prefix-list RED-ALLOWED seq 5 permit 10.1.0.0/16 le 24  

3. Render config on each leaf

   • Pull show running-config vrf RED and show route-map RED-IMPORT.
   • Compare line‑by‑line with intended.

4. Validate RT presence

   • show bgp vpnv4 vrf RED | include 65000:100
   • If missing on a leaf → RT import gap.

5. Validate filter presence/effect

   • show route-map RED-IMPORT
   • If absent or missing match ip address prefix-list RED-ALLOWED → filter gap.

6. Check BGP RIB vs. FIB

   • show bgp vpnv4 vrf RED (RIB)
   • show route vrf RED | include 10.2.3.0/24 (FIB)
   • If route present in RIB but missing from FIB → possible ASIC RT‑check failure (rare).

7. Data‑plane verification

   • On the suspect leaf, start a continuous ping with varying source ports to force different ECMP paths:

     for i in {1..100}; do ping -c 1 -s 1472 -p $(printf "%02x" $i) 10.2.3.8; done  

   • Capture on both leaves and inspect VNI tags.

8. Correlate with telemetry

   • Query flow records:

     SELECT * FROM netflow  
     WHERE src_ip='10.1.2.5' AND dst_ip='10.2.3.8' AND src_vrf!=dst_vrf;  

   • Non‑zero result confirms leak.

9. Remediate

   • Apply missing RT or filter via config push.
   • Verify that the leaf now shows the RT in BGP and the route‑map is applied.
   • Re‑run the ping test; leakage should cease.

10. Document drift

    • Record the diff between intended and observed config.
    • Add a test case to CI pipeline to prevent regression.

Common Pitfalls and Misconceptions