One virtual MAC typo and a week of weirdness

Introduction to Anycast Gateway Identity

Overview of Anycast Routing

Anycast routing assigns a single IP address to multiple forwarding nodes; packets are delivered to the nearest node according to the underlying unicast routing protocol (OSPF, IS‑IS, BGP). In a data‑center fabric the anycast address is typically the gateway IP for a tenant VLAN/VNI, hosted on every leaf switch that participates in the VXLAN overlay. The control plane (EVPN) advertises the anycast IP with a MAC address that is shared among all leaves; the data plane forwards traffic to the leaf whose underlay path has the lowest metric.

Importance of Consistent Virtual MAC

The virtual MAC (vMAC) is the layer‑2 identifier bound to the anycast IP in the overlay. When all leaves use the same vMAC:

• A host’s ARP entry resolves to one MAC regardless of which leaf it is attached to.
• Forwarding decisions in the underlay are based solely on IP routing; the overlay MAC does not influence path selection.
• Return traffic from the host follows the same reverse path as the forward traffic because the destination MAC seen by the host is invariant.
• Mobility detection (MAC‑move alarms) sees only genuine host movement, not apparent flapping caused by leaf‑specific MACs.

If the vMAC differs per leaf, the host learns multiple MACs for the same IP, breaking the symmetry assumptions above.

Understanding Intermittent Return‑Path Asymmetry

Causes of Return‑Path Asymmetry

Return‑path asymmetry occurs when the forward path (source → destination) and reverse path (destination → source) traverse different sequences of nodes. In an EVPN/VXLAN fabric asymmetry can be introduced by:

1. Unequal underlay metrics (e.g., ECMP hashing that selects different next‑hops for forward vs. reverse traffic).
2. Per‑leaf overlay state that causes the encapsulating VTEP to use a different destination MAC for the same anycast IP.
3. Asymmetric ARP/ND resolution where the host caches multiple MACs for the gateway IP.

The second cause is directly tied to an inconsistent virtual MAC: each leaf advertises its own vMAC for the anycast IP, so the host’s ARP cache may contain several entries. When the host sends a packet, it picks one MAC (often the first learned); the reply may be sent to a different MAC, causing the reply to be encapsulated by a different VTEP and thus travel a different underlay path.

Impact of Inconsistent Virtual MAC on Asymmetry

• Forward path: Determined by the MAC the host uses for the gateway. If the host picks leaf A’s vMAC, the packet is VXLAN‑encapsulated toward leaf A.
• Reverse path: The host’s reply uses the MAC it learned for the gateway from the ARP reply it received. If that reply came from leaf B (because leaf B responded to the ARP request), the host will use leaf B’s vMAC, directing the reply toward leaf B.
• The result is a micro‑loop where forward and reverse traffic traverse different leaves, producing intermittent asymmetry that correlates with ARP refresh cycles or load‑balanced ARP responses.
• Mobility alarms are triggered because the same host MAC appears to flap between VTEPs (leaf A and leaf B) each time the ARP cache changes, even though the host has not moved.

Observing Per-Leaf Config and Forwarding State

Configuring Per-Leaf Settings

In most vendor implementations the anycast gateway is configured under an IRB (Integrated Routing and Bridging) interface or a VLAN‑SVI that is then associated with a VNI. The virtual MAC is set either explicitly or via a system‑wide anycast‑gateway MAC pool.

Cisco NX‑OS (leaf)

interface Vlan101
  no shutdown
  ip address 10.0.101.1/24
  ipv6 address 2001:db8:101::1/64
  anycast-gateway
  anycast-gateway-mac 00:00.5e:00:01:01   ! explicit vMAC

Juniper Junos (leaf)

set vlans VLAN101 vlan-id 101
set interfaces irb unit 101 family inet address 10.0.101.1/24
set interfaces irb unit 101 family inet6 address 2001:db8:101::1/64
set routing-instances VPN101 instance-type virtual-switch
set routing-instances VPN101 interface irb.101
set routing-instances VPN101 vtep-source-interface lo0.0
set routing-instances VPN101 route-distinguisher 101:101
set routing-instances VPN101 vrf-target target:65100:101
set routing-instances VPN101 protocols evpn encapsulation vxlan
set routing-instances VPN101 protocols evpn extended-vni-list 10101
set routing-instances VPN101 protocols evpn default-gateway no-gateway   ! uses anycast-gw
set routing-instances VPN101 protocols evpn default-gateway mac-address 00:00.5e:00:01:01

Arista EOS (leaf)

interface Vlan101
   ip address 10.0.101.1/24
   ip address virtual 10.0.101.254/24   ! anycast IP
   ip virtual-router mac-address 00:00.5e:00:01:01

The intended state is the configuration above, identical on every leaf.

Monitoring Forwarding State

To verify that the applied state (what the forwarding plane actually uses) matches the intended state, inspect the MAC‑binding tables and the VXLAN encapsulation state.

Cisco NX‑OS

show mac address-table | include 00:00.5e:00:01:01
show nve peers
show nve vni 10101 peers
show forwarding distribution l2 multicast

Juniper Junos

show evpn database mac-address 00:00.5e:00:01:01
show interfaces irb.101 terse
show route forwarding-table destination 10.0.101.0/24

Arista EOS

show mac address-table address 00:00.5e:00:01:01
show vxlan vni 10101 remote-peers
show ip arp 10.0.101.254

The observed state is the output of these commands. A mismatch between the intended vMAC (e.g., 00:00.5e:00:01:01) and the observed MACs per leaf signals a configuration drift.

Comparing Intended and Observed Configurations

Identifying Inconsistencies in Virtual MAC

The comparison proceeds in three layers:

A typical drift scenario:

• Leaf A: anycast-gateway-mac 00:00.5e:00:01:01 (correct)
• Leaf B: command omitted → inherits the burned‑in MAC of the VLAN interface (e.g., 00:1c.73:00:0b:12)
• Result: Host learns two MACs for 10.0.101.1; ARP refresh alternates between them.

Troubleshooting Inconsistent Configurations

1. Verify rendered state on each leaf: compare the output of show running-config | include anycast-gateway-mac (NX‑OS) or equivalent.
2. Check applied state: confirm that the MAC appears in the L2 MAC table of the VLAN/VNI on all leaves.
3. Correlate with ARP logs: capture ARP requests/replies on a host to see which MAC is being returned.
4. Look for protocol‑specific overrides: some platforms allow a per‑VLAN MAC override that supersedes the anycast‑gateway MAC (e.g., Juniper’s mac-address under the IRB unit).

If a leaf shows a different MAC in the applied state but the rendered config matches the intent, the cause is likely a software bug or a hardware‑programming failure (e.g., MAC‑table write error). If the rendered config itself differs, the cause is a configuration drift (manual edit, failed orchestrator run, template mismatch).

Troubleshooting Intermittent Return-Path Asymmetry

Using CLI Commands to Identify Issues

A systematic CLI‑based workflow:

1. List per-leaf anycast‑gateway MACs

   # NX‑OS example, run on each leaf or via NX‑OS ansible
   show running-config | include anycast-gateway-mac

2. Validate MAC‑table symmetry

   show mac address-table | include 00:00.5e:00:01:01

   Expect the same MAC on every leaf; note any leaf that shows a different MAC or no entry.
3. Check EVPN MAC advertisements

   show bgp l2vpn evpn mac-address 00:00.5e:00:01:01

   Verify that the route originates from each leaf with the same MAC.
4. Capture ARP exchange on a host (Linux)

   sudo tcpdump -i eth0 -nn arp

   Observe whether ARP replies contain varying MAC addresses for the gateway IP.
5. Correlate with mobility alarms

   show logging log | include MAC-MOVE

   Timestamps should align with ARP reply changes.

Example Code: Debugging Virtual MAC Inconsistencies

Below is a Python script that logs into a set of NX‑OS switches via SSH, extracts the anycast‑gateway MAC, and reports inconsistencies.

#!/usr/bin/env python3
import netmiko

DEVICES = [
    {"host": "leaf01", "username": "admin", "password": "*****", "device_type": "cisco_nxos"},
    {"host": "leaf02", "username": "admin", "password": "*****", "device_type": "cisco_nxos"},
    # add more leaves as needed
]

def get_anycast_mac(conn):
    out = conn.send_command("show running-config | include anycast-gateway-mac")
    for line in out.splitlines():
        if "anycast-gateway-mac" in line:
            return line.split()[-1]
    return None

def main():
    macs = {}
    for dev in DEVICES:
        try:
            conn = netmiko.ConnectHandler(**dev)
            mac = get_anycast_mac(conn)
            macs[dev["host"]] = mac
            conn.disconnect()
        except Exception as e:
            print(f"Error connecting to {dev['host']}: {e}")
            continue

    unique = set(macs.values())
    if len(unique) == 1:
        print(f"All leaves consistent: {next(iter(unique))}")
    else:
        print("Inconsistency detected:")
        for host, mac in macs.items():
            print(f"  {host}: {mac}")

if __name__ == "__main__":
    main()

The script demonstrates the “why” (inconsistent vMAC leads to asymmetric paths) before showing the “how” (CLI retrieval).

Resolving Inconsistencies

• Configuration drift: Re‑apply the intended configuration via your orchestration tool and verify with show running-config.
• Software/hardware fault: Reload the affected line card or switch, or upgrade to a fixed firmware version if a known bug exists.
• Override detection: Remove any per‑VLAN MAC statements that conflict with the anycast‑gateway MAC, then re‑validate the MAC table.

After correction, the host will see a single gateway MAC, ARP replies will be symmetric, and mobility alarms will cease unless the host actually moves.