Design Review: Distributed Gateways and Stateful Inspection in EVPN Fabrics with Asymmetric Host‑to‑Host Paths

Author: Nadia Kareem – Routing Architect & Control‑Plane Specialist

Introduction to EVPN Fabrics

Overview of EVPN Technology

Ethernet VPN (EVPN) (RFC 7432) carries MAC/IP reachability and Ethernet segment information over BGP. Key NLRI types:

EVPN uses BGP for route distribution; data‑plane encapsulation can be VXLAN, MPLS, or SR‑v6.

Benefits of EVPN Fabrics

• Control‑plane‑only learning – eliminates flooding‑based MAC/IP learning.
• Multihoming & active‑active forwarding – ESI‑based DF election.
• Scalable L2/L3 convergence – BGP‑driven withdrawals.
• Policy‑rich routing – BGP communities, extended communities, route‑targets.
• Vendor‑agnostic – standard NLRI definitions.

Distributed Gateway Requirements

Gateway Functionality in EVPN Fabrics

A distributed (anycast) gateway provides Layer 3 routing between VNIs or VLAN‑based VRFs. Each leaf runs an IRB interface with the same anycast IP and MAC. Packet flow:

1. Host ARP/ND for the gateway IP.
2. Any leaf owning the anycast MAC replies.
3. Packet is encapsulated (VXLAN) toward the replying leaf (ingress).
4. Ingress leaf performs a VRF lookup, determines egress VNI/VLAN, and forwards to the egress leaf that owns the destination host’s MAC/IP (learned via EVPN Type‑2).

Thus, split across planes:

• Control plane – EVPN Type‑2 advertises the anycast gateway MAC/IP binding.
• Data plane – Ingress leaf does L3 lookup; packet may traverse the fabric to the egress leaf for final L2 encapsulation.

Stateful Inspection and Distributed Gateways

Stateful functions (firewall, NAT) require per‑flow state. In a distributed model the state must be present on the leaf that first sees the packet or be synchronized to the leaf that will see the return traffic.

• Asymmetric paths – Forward and reverse traffic may traverse different leaves (ECMP, load‑balanced spines, anycast gateway MAC load‑sharing). State must be available on both ingress and egress leaves, or return packets must be steered to the state‑holding leaf.
• State synchronization – Options: active‑active replication (dedicated sync protocol, VRRP‑like state sync, distributed DB), centralizing state in a service node with service chaining, or using stateless techniques (deterministic source‑NAT).
• Failover consistency – Leaf failure requires rapid state recovery to avoid connection loss.

Asymmetric Host‑to‑Host Paths in EVPN Fabrics

Asymmetry is normal when:

• ECMP across multiple spines for VXLAN encapsulation/decapsulation.
• Anycast gateway MAC load‑balanced via LAG/VPC across leaves.
• Optimized routing (symmetric vs. asymmetric IRB) selects egress leaf by destination location while ingress leaf is chosen by source ARP resolution.

Consequently, forward path may be Leaf A → Spine X → Leaf B while reverse path is Leaf C → Spine Y → Leaf A. Unsynchronized stateful functions see only one direction, causing drops or NAT mis‑bindings.

NAT and Firewall State Placement

Placement Options

Impact of Asymmetric Paths

• Centralized – Requires bidirectional steering to the same service instance; needs symmetric hashing (5‑tuple) or service‑node affinity (ECMP‑aware load balancer). If ECMP hash varies per direction, reverse packets may hit a different instance → state loss.
• Distributed – Tolerable only if state is shared/mirrored between ingress and egress leaves; otherwise the leaf seeing only one direction drops packets or creates incorrect NAT bindings.
• Hybrid – Stateless NAT works regardless of asymmetry; centralized inspection still faces the same symmetry requirement as the pure centralized model.

Comparison of State Placement Approaches

Design Considerations for Distributed Gateways

Gateway Deployment Models

1. Anycast IRB (Symmetric) – Each leaf configures the same anycast gateway IP/MAC. Hosts ARP to the nearest leaf (underlay ECMP). Forward/reverse paths may differ, but gateway function is identical on all leaves.
2. Optimized IRB (Asymmetric) – Leaf advertises a distinct gateway IP per VRF (or per ESI) while still advertising the anycast MAC for L2 forwarding. Ingress leaf chosen by source host; egress leaf selected by destination location (often reduces tromboning).
3. Centralized GW – A pair of spine/super‑spine nodes hosts the IRB; leaves perform only L2 bridging. Eliminates per‑leaf routing tables but introduces hair‑pin for all inter‑subnet traffic.

For stateful services, anycast IRB is most common because it keeps the gateway function locally available, though it amplifies asymmetry. Optimized IRB can reduce asymmetry if the fabric ensures the same leaf is chosen for both directions (e.g., source‑and‑destination hash symmetry in the underlay).

State Synchronization and Consistency

• Protocol choices –
  • VRRP/HSRP‑like state sync – hello‑based exchange of connection‑tracking tables; suitable for small fabrics.
  • Distributed database (etcd, Consul) – leaves act as clients; state updates via RPC; strong consistency but added latency.
  • Multicast‑based state sync – leverages existing EVPN BUM flood (Type‑3) to disseminate state deltas; requires careful sequencing.
• Consistency model – NAT can tolerate eventual consistency if NAT timeout exceeds sync latency (e.g., 30 s timeout, sync < 5 s). Firewalls that drop packets on missing state need strong consistency (or at least read‑after‑write) to avoid asymmetric drops.
• Conflict resolution – When two leaves simultaneously learn a new flow (e.g., SYN from host A to host B via different leaves), they must agree on a single NAT translation. Strategies:
  • Deterministic allocation – derive NAT port from a hash of the 5‑tuple plus a leaf‑specific salt; both leaves compute the same mapping without communication.
  • Owner election – use ESI or lowest leaf IP as flow owner; other leaves forward packets to the owner via encapsulation (service chaining).

Scalability and Performance Considerations

• State size – Approx. 200 bytes for TCP connection‑tracking + 64 bytes for NAT binding. A leaf handling 1 M concurrent flows needs ~250 MB RAM.
• TCAM/ACL impact – Stateful inspection via ACL‑based packet marking + service lookup may require additional TCAM entries for match‑action rules.
• Sync bandwidth – State delta traffic scales with flow churn rate. For 10 k leaf nodes each seeing 10 k new flows/s, sync stream can reach hundreds of Mbps; a dedicated sync VLAN or overlay is advisable.
• CPU – Packet‑level state lookups add ~50‑150 ns per packet on modern ASICs. Offload to NPUs or use flow‑based caching (SRAM flow table) to mitigate impact.
• Failover time – State replication must converge faster than NAT/firewall idle timeout to avoid premature flow termination. Target: < 1 s for TCP, < 500 ms for UDP.

Troubleshooting Distributed Gateway Issues

Common Issues

End of review.