Introduction to EVPN MAC Mobility

Overview of EVPN and MAC Mobility

Ethernet VPN (EVPN) defined in RFC 7432 carries MAC address reachability information in the control plane using BGP‑NLRI of type 2 (MAC/IP Advertisement Route). When a host moves from one Ethernet Segment (ES) to another, the originating Provider Edge (PE) advertises a new MAC/IP route with an incremented MAC Mobility Extended Community. The sequence number in that community allows remote PEs to detect whether the advertisement represents a new location or a stale duplicate.

The MAC Mobility Extended Community (type 0x06) encodes:

• Flags (including the Sticky bit)
• Sequence Number (4‑byte unsigned integer)
• MAC Address (6 bytes) – carried in the NLRI itself

When a PE receives a MAC/IP route for a MAC that it already knows, it compares the incoming sequence number with the locally stored one:

• If incoming > local, the route is accepted as a valid move and the local entry is updated.
• If incoming = local, the route is treated as a duplicate and ignored.
• If incoming < local, the route is considered outdated and is discarded (or may trigger a withdrawal depending on implementation).

Importance of Sequence Numbering in EVPN

The sequence number provides a total order for MAC moves without relying on timestamps, which can be skewed across devices. It enables:

• Loop‑free convergence: older advertisements are automatically ignored.
• Mobility detection: a rapid increase in sequence number signals frequent moves (potential flapping).
• Sticky MAC handling: when the Sticky bit is set, the MAC is expected to remain attached to the originating ES; any move triggers a mobility withdrawal.

Correct operation hinges on all PEs interpreting the sequence number consistently and on the guarantee that a move is advertised once per transition. Any break in that guarantee—such as partial isolation or inconsistent handling—can cause the control plane to oscillate, producing a MAC mobility storm.

Understanding Partial Isolation Failures

Definition and Causes of Partial Isolation Failures

A partial isolation failure occurs when a PE loses bidirectional connectivity to a subset of its EVPN peers while maintaining connectivity to others. Unlike a total isolation (where the PE loses all BGP/EVPN sessions), partial isolation leaves some BGP peers active, allowing the PE to continue sending and receiving routes for a portion of the fabric.

Common causes:

Impact of Partial Isolation Failures on EVPN Networks

When a PE is partially isolated:

1. Withdrawals may not reach all peers – the PE may withdraw a MAC/IP route (e.g., on local link down) but only a subset of remote PEs receive the withdrawal.
2. Advertisements may be sent to only a subset – the PE may re‑advertise the MAC with an incremented sequence number, but only the reachable peers process it.
3. Resulting state divergence – some PEs believe the MAC is at location A (old sequence), others believe it is at location B (new sequence). The inconsistency triggers repeated mobility updates as each PE attempts to reconcile the conflict.

The control‑plane trace:

• State change origin: Local link event (e.g., host moves or link flaps) triggers a MAC/IP route withdrawal and a new advertisement with incremented sequence number from the originating PE.
• Propagation: The withdrawal reaches only the connected subset of peers; the new advertisement reaches the same subset (or a different subset if asymmetry exists).
• Consumption: Peers that received both withdrawal and advertisement update their MAC table to the new location and sequence number. Peers that received only one of the two retain stale state.
• Forwarding divergence: Packets destined for the MAC are forwarded according to each PE’s local MAC table, leading to hair‑pinning, black‑holing, or looping until the inconsistency is resolved—or until the sequence number keeps incrementing, causing a storm.

Identifying Partial Isolation Failures in EVPN Environments

Operators should look for asymmetric BGP/EVPN session states and mismatched route counts:

• Show BGP summary – note sessions that are Established but have Received/Prefix counters that do not increase over time.
• Show EVPN database – compare the number of MAC/IP routes learned from each peer; a significant discrepancy suggests selective filtering.
• Monitor interface counters – look for rising RX errors or drops on the physical/link layer while TX remains healthy.
• Use BGP graceful‑restart or BGP‑monitoring protocols (BMP) to verify that update messages are being transmitted and received symmetrically.
• Correlate with syslog – messages like %BGP-5-ADJCHANGE: neighbor X.X.X.X Down appearing only on a subset of peers.

Inconsistent Sequence Handling in EVPN

Sequence Numbering Mechanisms in EVPN

Per RFC 7432, the MAC Mobility Extended Community carries a 4‑byte sequence number that is incremented by the originating PE each time the MAC address changes Ethernet Segment. The sequence number space is modulo 2³²; wraparound is allowed but must be handled correctly.

Key points:

• The originating PE increments the sequence number before advertising the new MAC/IP route.
• The PE does not increment the sequence number when merely refreshing an existing advertisement (unless a move actually occurred).
• Remote PEs compare the incoming sequence number to the locally stored one using unsigned integer arithmetic (i.e., treat higher numeric value as newer, accounting for wrap‑around using standard serial number comparison).

Consequences of Inconsistent Sequence Handling

If any PE deviates from the RFC‑defined comparison logic, the following can occur:

These inconsistencies turn a single mobility event into a sequence number oscillation: each PE’s inconsistent handling causes it to reject the advertisement, generate its own withdrawal with an (incorrectly) incremented sequence, and the cycle repeats across the fabric.

Troubleshooting Inconsistent Sequence Handling Issues

1. Verify the sequence number values in the MAC Mobility Extended Community on both the advertising and receiving PEs.
   • On Juniper: show route evpn mac <mac-address> extensive
   • On Cisco IOS‑XR: show bgp l2vpn evpn route-detail <rd>:<mac>
   • On Arista EOS: show bgp evpn mac <mac>
2. Check the local stored sequence number (often displayed alongside the MAC entry).
3. Correlate timestamps – if the sequence number increments faster than the expected mobility rate (e.g., >1 move per second), suspect inconsistent handling.
4. Look for sticky‑bit mismatches – a MAC marked sticky on one PE but not another can cause the receiving PE to treat each receipt as a new move.
5. Enable BGP debug for EVPN updates (vendor‑specific, use sparingly) to see the exact extended community values being transmitted and received.
6. Compare software versions – known bugs in sequence‑number comparison have been fixed in recent releases (e.g., Juniper Junos 20.4R3, Cisco IOS‑XR 7.5.2).

EVPN MAC Mobility Storms

Characteristics of MAC Mobility Storms

A MAC mobility storm manifests as:

• Rapid, repeated increments of the MAC Mobility sequence number for a given MAC (often reaching the maximum 2³²‑1 within minutes).
• High CPU utilization on the PE’s BGP/EVPN process due to continuous route processing.
• Frequent MAC table churn – the forwarding information base (FIB) is constantly updated, causing packet loss, out‑of‑order delivery, and increased latency.
• Log messages indicating “MAC move detected” or “MAC mobility dampening triggered” at a high rate (e.g., >10 moves/sec).
• No corresponding physical host movement – the host remains attached to a single ES, or moves only occasionally.

The storm is a control‑plane feedback loop: each PE’s inconsistent handling generates a new withdrawal/advertisement, which triggers the same reaction in peers, causing the sequence number to climb without bound.

Distinguishing Between Host Flapping and MAC Mobility Storms

Case Studies of MAC Mobility Storms in EVPN Networks

Case 1: Unidirectional Link Loss Between Two Data‑Center Pods

• Topology: Pod A (PE‑A1, PE‑A2) ↔ Pod B (PE‑B1, PE‑B2) via two parallel 100 G links.
• Failure: TX direction of link A1‑B1 failed; RX direction remained functional, keeping BGP sessions ESTABLISHED.
• Sequence: Host H1 attached to ES‑1 in Pod A moved to ES‑2 in Pod B (valid move). PE‑A1 sent withdrawal + new advertisement; only PE‑B2 received both (due to RX‑only path). PE‑B1 received only the withdrawal (TX‑lost) and therefore retracted the MAC, sending its own withdrawal with an incremented sequence number. PE‑A2, having received only the advertisement, re‑advertised with yet another increment. The sequence number bounced between pods, causing a storm.
• Resolution: Fixing the unidirectional link restored symmetric update exchange; sequence number stabilized.

Case 2: ACL Mis‑Match on Route‑Target Filtering

• Topology: Three PEs (PE‑1, PE‑2, PE‑3) import/export RT 65000:100.
• Failure: An ingress ACL on PE‑2 incorrectly denied BGP updates carrying RT 65000:100 only when the update’s NLRI contained a MAC address learned from PE‑3.
• Sequence: Host H2 moved from ES‑A (PE‑1) to ES‑B (PE‑3). PE‑3 advertised the new MAC/IP route; PE‑1 received it and updated. PE‑2 dropped the advertisement due to the ACL, so it retained the old sequence number. When PE‑2 later sent a periodic refresh (or a withdrawal due to a local timer), PE‑1 and PE‑3 saw a lower sequence number and ignored it, while PE‑2 kept re‑advertising with an incremented number, causing a loop.
• Resolution: Correcting the ACL restored symmetric advertisement exchange; the sequence number stopped climbing.

Coordination Problems Between Domains

Inter‑Domain Coordination in EVPN

EVPN can span multiple administrative domains (e.g., data‑center pods managed by different teams, or a campus core interconnecting several sites). Coordination relies on:

• Uniform route‑target (RT) import/export policies ensuring that MAC/IP routes are exchanged across domain borders.
• Consistent MAC Mobility Extended Community handling (sequence number comparison, sticky‑bit treatment) across all domain edge devices.
• Synchronized BGP timers (hold time, keepalive) to avoid transient session asymmetry that could masquerade as partial isolation.
• Shared MAC mobility dampening parameters (if used) to prevent one domain from interpreting a move as a flap while another does not.

When any of these elements diverge, the domains develop different views of the same MAC’s location, leading to cross‑domain mobility storms.

Challenges in Achieving Seamless Inter-Domain Coordination