When STP IGP and BGP Recovered in the Wrong Order

Introduction to the Event

Background and Context

The incident occurred in a three‑tier leaf‑spine data‑center fabric that provides both Layer‑2 (VXLAN‑based overlay) and Layer‑3 (OSPF underlay, BGP EVPN overlay) services. The fabric consists of:

Leaf switches – model‑agnostic, running a modern NOS; host servers, provide MLAG/vPC peer‑links, and terminate VXLAN VTEPs.
Spine switches – pure Layer‑3 connectivity; run OSPF in the underlay and MP‑BGP for EVPN route exchange.
Centralized PDU feeding each rack; an upstream feeder tripped, causing a simultaneous loss of AC power to an entire rack of leaf switches and two spine switches.

The design intent is that, after power restoration, each control plane (Layer‑2 and Layer‑3) should reconverge independently and the combined forwarding plane should resume normal traffic flow within the sum of their individual convergence times.

Event Timeline and Key Milestones

Time (UTC)	Milestone
T‑00:00	Normal operation; all links up, STP stable, LACP aggregators active, OSPF adjacencies full, BGP EVPN sessions established.
T+00:00	PDU breaker trips; instantaneous loss of power to all affected switches.
T+00:05	Power restored by UPS/generator; switches begin boot sequence.
T+00:30	All switches have completed boot and initialized hardware.
T+00:45	Layer‑2 control planes (LACP, STP/MSTP, MLAG peer‑link) report link‑up and begin reconvergence.
T+00:50	OSPF hellos begin exchanging on interfaces that are administratively up but whose underlying LACP aggregator is not yet marked “collecting/distributing”.
T+01:00	OSPF adjacencies reach FULL state on several leaf‑spine port‑channel interfaces (per OSPF log).
T+01:10	BGP EVPN sessions re‑establish using the OSPF‑learned underlay routes.
T+01:20	STP/MSTP has elected a new root and blocked redundant ports; LACP aggregators on leaf‑spine links are still negotiating.
T+01:30	Traffic loss observed: ping/TCP flows between servers attached to different leaves experience >90% packet loss.
T+02:00	LACP aggregators finally transition to collecting/distributing state; LACP logs show “LAG interface up”.
T+02:05	Forwarding plane resumes normal traffic; packet loss drops to <1%.
T+02:30	All control planes report stable state; no further alarms.

Key observation: Layer‑2 (LACP, STP) and Layer‑3 (OSPF, BGP) each reported successful recovery on their own timelines, yet the combined convergence produced a ~75‑second window of traffic blackhole.

Technical Overview of the Infrastructure

Layer‑2 Network Architecture

Physical links: 10 GbE SFP+ between leaf and spine, bundled into LACP port‑channels (active‑active).
LACP: Configured in active mode on both ends; lacp rate fast (1 s packet interval) where supported, otherwise default slow (30 s).
STP/MSTP: IEEE 802.1s MSTP with two instances (instance 0 for VLAN 1‑1000, instance 1 for VLAN 1001‑2000). Hello time 2 s, forward delay 15 s, max age 20 s (standard values).
MLAG/vPC: Peer‑link formed with dual‑homed leaf pairs; keepalive via dedicated management VLAN.
VXLAN: VTEPs located on leaf switches; flood‑and‑learn for unknown unicast, head‑end replication for known unicast/multicast.
Layer‑2 control protocols: LACP (802.3ad), STP/MSTP (802.1w/s), LLDP (optional), and MLAG keepalive (proprietary).

Layer‑3 Network Architecture

Underlay routing: OSPFv2 (RFC 2328) running on all leaf and spine switch loopback interfaces (OSPF router‑ID = loopback IP). Physical LACP port‑channels are placed in OSPF area 0 as point‑to‑point links.
OSPF timers: Hello interval 10 s, dead interval 40 s (RFC 2328 defaults). No OSPF BFD enabled in baseline.
Inter‑area: None; flat area 0 design.
Overlay routing: MP‑BGP (RFC 4760) EVPN address‑family (type‑2 MAC‑IP routes, type‑3 inclusive multicast routes). BGP hold time 180 s, keepalive 60 s (default).
Route redistribution: None; underlay provides reachability for BGP next‑hops.
Layer‑3 control protocols: OSPF, BGP, ICMP (for ping/trace), and optionally BFD (not enabled).

Interdependencies and Interactions Between Layers

LACP → OSPF: OSPF treats the LACP port‑channel as a single logical interface. OSPF adjacency formation depends only on the logical interface being administratively up and having an IP address; it does not wait for the LACP aggregator to reach the collecting/distributing state.
STP → VXLAN: STP port state (blocking/forwarding) determines whether the underlying physical port can carry VXLAN encapsulated traffic. If a port is blocked, VXLAN packets are dropped at the ingress of the physical port.
MLAG/vPC → LACP: The peer‑link LACP aggregator must be up for the MLAG peer‑sync to exchange MAC tables and sequence numbers; otherwise the peer‑link remains in a standby state, causing asymmetric MAC learning.
BGP → OSPF: BGP EVPN next‑hop resolution relies on OSPF‑installed underlay routes; if OSPF installs a route over a not‑yet‑ready LACP link, BGP will program that next‑hop into the FIB, leading to blackhole until the LACP link becomes usable.

These dependencies mean that a timing mismatch between Layer‑2 link readiness and Layer‑3 adjacency can cause a transient forwarding blackhole even though each control plane believes it has converged.

The Power Event and Initial Response

Description of the Power Event

At T+00:00 a feeder breaker supplying the rack’s PDU tripped, causing an instantaneous loss of AC power to:

Four leaf switches (Leaf‑1 … Leaf‑4) in the rack.
Two spine switches (Spine‑A, Spine‑B) that shared the same PDU.

All affected switches experienced a hard power loss; line cards, supervisors, and fans went offline. No graceful shutdown was possible.

Immediate Effects on Layer‑2 and Layer‑3 Controls

Layer‑2: All LACP port‑channels went to detached state; STP ports transitioned to blocking (due to loss of BPDU). MLAG peer‑links went down, causing the peer‑keepalive mechanism to detect loss of peer.
Layer‑3: OSPF interfaces lost IP connectivity; OSPF neighbors went to Down. BGP EVPN peers lost TCP connectivity and entered Idle state. Loopback interfaces remained up (powered via internal backup) but were isolated because physical links were down.

Initial Troubleshooting and Recovery Efforts

Operations team performed the following steps immediately after power restoration:

Verified PDU output and UPS status.
Checked switch console logs for boot completion (show version, show system uptime).
Confirmed that all switches had re‑initialized line cards and that link LEDs were amber/green as expected.

Initiated Layer‑2 health checks:

show lacp neighbor
show spanning-tree summary

Initiated Layer‑3 health checks:
```
show ip ospf neighbor
show bgp summary
```

At T+00:45 the team observed that LACP and STP reported link‑up and began exchanging protocol packets; OSPF hellos were seen on the wire, but no adjacencies had yet reached FULL.

Layer‑2 Control Recovery

Successful Recovery Mechanisms

LACP: Each end transmitted LACP packets at the configured rate (fast/slow). After missing three consecutive LACP PDUs, the partner considered the link down and moved the aggregator to a detached state. Once power was restored, both sides exchanged LACP PDUs; after the required number of successful handshakes (typically three), the aggregator transitioned to collecting/distributing state.
STP/MSTP: With the physical links up, BPDUs resumed. The protocol recalculated the spanning tree, elected a new root bridge, and moved previously blocking ports to forwarding after the standard forward‑delay timers (2 × forward delay + hello time).
MLAG/vPC: The peer‑link LACP aggregator came up, allowing the keepalive mechanism to re‑establish peer synchronization. MAC tables and sequence numbers were exchanged, restoring symmetric learning.

By T+02:00 all LACP aggregators on leaf‑spine links showed LAG interface up, and STP/MSTP had a stable topology with no blocked ports affecting VXLAN traffic.

Layer‑3 Control Recovery

OSPF*

OSPF:

Layer‑3 Control Recovery

OSPF Reconvergence

OSPF hellos began at T+00:50 on interfaces that were administratively up. Because OSPF does not wait for LACP collecting/distributing, adjacencies reached FULL state as early as T+01:00 on several leaf‑spine port‑channels, even though the underlying links were not yet ready to forward data.
OSPF installed routes over these adjacencies into the RIB and subsequently into the FIB.

BGP EVPN Reconvergence

With OSPF routes in place, BGP EVPN sessions re‑established at T+01:10 using the OSPF‑learned underlay next‑hops.
BGP advertised EVPN type‑2 (MAC‑IP) and type‑3 (multicast) routes, which were installed in the overlay FIB.

Resulting Forwarding State

At T+01:30 the data plane began forwarding packets based on the newly installed FIB entries.
However, because the underlying LACP links were still in negotiating or detached state, packets were blackholed at the physical layer, resulting in >90% packet loss for inter‑leaf traffic.
Once LACP reached collecting/distributing at T+02:00, the physical ports could carry traffic, the FIB entries became valid, and normal forwarding resumed by T+02:05.

Combined Convergence Failure Analysis

Factor	Layer‑2 Timeline	Layer‑3 Timeline	Impact
LACP collecting/distributing	T+02:00	—	Physical link not usable before this time.
OSPF adjacency FULL	—	T+01:00	Routes installed over not‑yet‑ready links.
BGP EVPN session up	—	T+01:10	Overlay routes programmed with blackhole next‑hops.
STP/MSTP stable	T+01:20	—	No Layer‑2 blocking after this point.
Traffic loss observed	T+01:30	—	>90% packet loss due to LACP not ready.
Forwarding restored	T+02:05	—	LACP ready, traffic flows.

The root cause is the lack of synchronization between OSPF/BGP convergence and LACP link readiness. OSPF and BGP consider the port‑channel “up” as soon as the logical interface is administratively active, ignoring the LACP state machine. Consequently, the control plane converges faster than the data plane can support, creating a transient blackhole.

Mitigation and Recommendations

Enable OSPF BFD on LACP port‑channels – BFD will detect forwarding failures faster than OSPF dead intervals and prevent adjacency formation over not‑ready links.
Delay OSPF adjacency until LACP reports collecting/distributing – Some NOSs support a lacp delay or ospf wait-for-lacp knob; enable it if available.
Tune LACP timers – Use lacp rate fast uniformly across the fabric to reduce the time to reach collecting/distributing.
Deploy MLAG peer‑link health checks – Ensure the peer‑link LACP is up before allowing vPC/MLAG to forward traffic.
Monitor and alert on LACP state – Create alerts for any port‑channel that remains in detached or negotiating state beyond a threshold (e.g., 30 s).
Validate design with failure‑injection testing – Simulate power loss and measure end‑to‑end convergence; adjust timers accordingly.

Implementing these changes will align Layer‑2 and Layer‑3 convergence times, eliminating the overlapping window that caused the traffic blackhole.

Conclusion

The power event exposed a subtle timing dependency: Layer‑3 protocols (OSPF, BGP) converged before Layer‑2 link aggregation reached a usable state. Although each control plane independently reported success, the mismatch produced a ~75‑second forwarding blackhole. By tightening the coupling between LACP readiness and