Correlating MAC churn with link flaps and FDB age-outs

Introduction to Telemetry and Event Artifacts

Telemetry provides continuous, time‑stamped network state; event artifacts (logs, traps, CLI snapshots) give discrete context. In an EVPN‑VXLAN fabric, the most useful telemetry streams are gNMI (or streaming telemetry) exports of OpenConfig or vendor YANG models.

Key metric families

Event artifacts to correlate with these metrics include syslog/%LINK‑3‑UPDOWN, SNMP traps, streaming alerts, trace spans, and ad‑hoc CLI show output.

Underlay Link Instability

Detecting Flaps with CLI

# Cisco Nexus leaf – error counters
show interface ethernet1/1 counters errors

Sample output

Eth1/1:
  InErrors: 0          OutErrors: 0
  InDiscards: 0        OutDiscards: 0
  Last link flapped: 00:12:34 ago (2 flaps in last 5 min)

A non‑zero flap count or alternating %LINK-3-UPDOWN syslog messages indicate instability.

If the switch lacks a flap counter, derive it from syslog:

show logging logfile | include "%LINK-3-UPDOWN"

Telemetry‑Based Flap Detection

Assuming OpenConfig ifOperStatus exported via gNMI:

# State transitions per interface in the last 5 min
sum by (interface) (
  changes(ifOperStatus[5m])
) > 5

Interfaces exceeding the threshold are flapping.

To measure down time:

# Total seconds interface was down in the last 15 min
sum by (interface) (
  (1 - ifOperStatus) * on() group_left() time()
)[15m:]

High “down‑seconds” suggest persistent outages or rapid flaps.

Event Correlation

Link flaps generate syslog (%LINK-3-UPDOWN) and, if enabled, SNMP linkDown/linkUp traps. Aligning a PromQL spike with these logs confirms the telemetry reflects real hardware events.

Blind spots

• Polling intervals > sub‑second flaps miss events.
• Error counters may not increment for marginal SFP issues.
• Lack of per‑lane optical power telemetry prevents distinguishing transceiver drift from ASIC faults.

Minimum telemetry set

1. ifOperStatus (prefer ON_CHANGE subscription).
2. ifInErrors / ifOutErrors.
3. Syslog or SNMP trap for linkDown/linkUp (validation).

EVPN MAC Route Churn

CLI‑Based Churn Inspection

# View a specific MAC route
show bgp evpn mac address-table | include "00:11:22:33:44:55"

Sample

Network          Next Hop            Metric LocPrf Weight Path
*> 00:11:22:33:44:55/48  10.0.0.2               0    100      0 65000 i
   Last update: 00:00:03 ago
   Age: 00:00:03

Repeated runs show a rapidly changing Last update field.

Many platforms expose a MAC‑move counter:

show evpn mac-move statistics

Sample

MAC               Moves (last 5min)  Last move time
00:11:22:33:44:55 27                 00:00:12

A high “Moves” count (>10/min) signals churn.

Telemetry‑Based Churn Detection

If the exporter includes a mac label:

# MAC route update rate (updates/sec) per leaf
rate(evpnMacRouteUpdates_total[1m])

Values far above baseline (e.g., >5 updates/sec) indicate churn.

To identify offending MACs:

# Top 5 MACs by update rate in the last 2 min
topk(5, sum by (mac) (rate(evpnMacRouteUpdates_total[2m])))

Without a mac label, only aggregate churn is visible; CLI or logs are needed to pinpoint the source.

Event Artifacts

MAC moves generate syslog such as %EVPN-5-MAC_MOVE:

show logging logfile | include "%EVPN-5-MAC_MOVE"

Sample

%EVPN-5-MAC_MOVE: MAC 00:11:22:33:44:55 moved from VNI 10100 to VNI 10101 (seq 57)

Repeated messages for the same MAC confirm churn.

Blind spots

• Aggregate evpnMacRouteUpdates_total hides per‑MAC details.
• Platforms may suppress MAC‑move syslog after a threshold.
• Missing per‑VNI MAC metrics prevents distinguishing VNI misconfiguration from host mobility.

Minimum telemetry set

1. evpnMacRouteUpdates_total with mac (or at least vni) label.
2. bgpPeerState to rule out peer flaps.
3. Syslog for %EVPN-5-MAC_MOVE (or equivalent).

Local Forwarding Database (FDB) Aging

CLI Inspection of the FDB

show mac address-table

Sample

          Mac Address Table
-------------------------------------------
VLAN    MAC Address       Type        Ports
----    -----------       -----       -----
10      00:11:22:33:44:55 Dynamic     Et1/1
20      66:77:88:99:AA:BB Dynamic     Po5
30      CC:DD:EE:FF:00:11 Static      Et2/2

Configured aging timer:

show mac address-table aging-time

Sample

Global Aging Time: 300 seconds

Entry‑specific age (if supported):

show mac address-table address 00:11:22:33:44:55 detail

Sample

MAC Address: 00:11:22:33:44:55
    VLAN: 10
    Port: Et1/1
    Type: Dynamic
    Age: 45 seconds

Ages consistently near the timeout indicate normal churn; ages far below the timer with steady traffic suggest overly aggressive aging.

Telemetry‑Based Aging Analysis

Assuming per‑entry age exported as a histogram (fdbEntryAgeSeconds_bucket):

# Fraction of FDB entries older than 250 s (default 300 s timeout)
sum by (device) (
  increase(fdbEntryAgeSeconds_bucket{le="250"}[5m])
) /
sum by (device) (
  increase(fdbEntryAgeSeconds_count[5m])
)

A rising fraction (>0.8) shows many entries close to expiration; combined with stable traffic, this may point to premature aging or excessive flushing.

Blind spots

• Without per‑entry age granularity, only aggregate learn/age counters are available, limiting root‑cause insight.
• Some platforms do not export FDB age at all, requiring reliance on CLI show mac address-table output.

Minimum telemetry set

1. fdbEntryAgeSeconds (gauge or histogram) – preferably with a MAC label.
2. fdbLearningTotal and fdbAgingTotal counters (to correlate learns vs. ages).
3. fdbFloodsTotal (indicator of unknown‑MAC flooding due to premature aging).

By aligning these telemetry streams with corresponding event artifacts and targeted CLI checks, operators can distinguish genuine fabric healing from superficial symptom masking.