Refactoring brittle ECMP policy into deterministic failover

Introduction to Fabric Refactoring

In a leaf‑spine fabric, equal‑cost multipath (ECMP) forwarding often arises accidentally when the IGP or BGP computes identical path metrics to a destination across multiple spines. This emergent behavior depends on perfect symmetry—identical link costs, spine counts, and IGP areas—and on uniform path advertisement to every leaf. When symmetry breaks due to mis‑configured costs, spine failure, or asymmetric policy, the set of equal‑cost paths can shift unpredictably, causing traffic to move or drop without explicit operator intent.

Refactoring replaces this accidental symmetry with three explicit mechanisms:

• Explicit backup intent – a declarative primary/backup relationship.
• Stable recursion – deterministic next‑hop resolution that changes only on true loss.
• Observable failover checkpoints – measurable signals that confirm backup activation.

Understanding Equal‑Cost Behavior

ECMP installs all next‑hops with the same weight when the routing protocol reports identical metrics. The forwarding plane spreads traffic according to a hash algorithm. Because the equal‑cost set is derived from IGP metrics, operators cannot express service‑specific preferences without manipulating those metrics, which is opaque and error‑prone.

Accidental Path Symmetry and Its Limitations

Accidental symmetry occurs when the fabric presents the same number of equal‑cost routes to each leaf, but the underlying reasons are undocumented or unenforced.

• Hidden fragility – a single link‑cost change can break symmetry, leaving some leaves with a single path and increasing congestion risk.
• Policy opacity – expressing “prefer spine A for service X, spine B for service Y” requires metric tweaks that are hard to track.
• Observability gap – no explicit signal shows when a leaf uses a backup path; only flow‑hash distribution changes, which is difficult to correlate with a failure.
• Failure‑domain bleed – spine failure can trigger transient micro‑loops or sub‑optimal paths until IGP converges, and the resulting path may not match the intended backup design.

These issues motivate the refactor toward explicit backup intent, stable recursion, and observable failover checkpoints.

Identifying Requirements for Explicit Backup Intent

Defining Backup Intent and Its Benefits

Backup intent is a declarative rule: “If the primary next‑hop for prefix P becomes unreachable, install next‑hop Q as the backup.” Unlike ECMP, which treats all equal‑cost paths as interchangeable, backup intent creates a deterministic primary/secondary relationship that is:

• Deterministic – the backup is known a priori and does not depend on IGP metric fluctuations.
• Policy‑driven – can be tied to route‑targets, VRFs, or service‑level objectives without altering underlay metrics.
• Visible – the RIB can flag a route as “primary” or “backup,” enabling direct monitoring.

Benefits include faster, predictable failover; simplified capacity planning (backup paths can be sized independently); and the ability to enforce service‑specific routing (e.g., low‑latency traffic prefers spine 1, best‑effort prefers spine 2).

Stable Recursion in Fabric Design

Recursion resolves a BGP next‑hop (e.g., a spine loopback) to an outgoing interface via the IGP. Stable recursion means the resolved path changes only when the underlying next‑hop reachability changes, remaining loop‑free and deterministic.

Key properties:

• No churn on IGP metric changes – if the IGP metric to a spine changes but the spine stays reachable, the recursive next‑hop for the BGP route remains the same.
• Fast convergence on true loss – when the spine becomes unreachable, the recursion withdraws the BGP next‑hop promptly, allowing the backup intent to install.
• FIB consistency – the resolved next‑hop is programmed into the FIB exactly as the RIB intends, avoiding RIB/FIB divergence.

Observable Failover Checkpoints for Reliability

A failover checkpoint is a measurable point in the control‑plane/data‑plane path that confirms whether the intended backup is active. Examples:

• RIB flag – a community or attribute marking a route as backup.
• FIB entry – presence of a specific next‑hop label or ECMP weight.
• Telemetry stream – periodic export of the next‑hop selection decision (e.g., via gNMI or streaming telemetry).
• Control‑plane log – log entry when a backup route is installed or withdrawn.

These checkpoints give operators an explicit, observable signal that the fabric behaves as designed, enabling automated validation and reducing mean‑time‑to‑detect (MTTD) for failures.

Refactoring the Fabric

Step 1: Assessing Current Fabric Topology

Before any changes, collect:

1. Topology diagram – leaf‑spine connections, link costs, existing IGP areas.
2. IGP metric distribution – show isis database or show ospf database to verify symmetry.
3. BGP multipath settings – show bgp neighbors <peer> received-routes to see advertised path counts.
4. Current ECMP hash – show forwarding-table <prefix> to view active next‑hops.
5. Failure‑mode logs – recent logs of spine/link flaps and observed traffic shifts.

Compare the observed ECMP count to the designed number of equal‑cost paths. If they match only by coincidence (e.g., all links have cost 10 but no policy enforces it), mark the fabric as “accidentally symmetric.”

Step 2: Implementing Explicit Backup Intent

Example Configuration (vendor‑neutral route‑policy)

# Define communities
community-set PRIMARY_SPINE  members 65000:100
community-set BACKUP_SPINE   members 65000:200

# Route policy to attach community based on source
route-policy MARK_SPINE
  if source-address in (10.0.0.1/32) then   # primary spine loopback
    set community PRIMARY_SPINE
  elseif source-address in (10.0.0.2/32) then # backup spine loopback
    set community BACKUP_SPINE
  else
    pass
  endif
end-policy

# Apply policy inbound from spine peers
router bgp 65000
  neighbor 10.0.0.1
    route-policy MARK_SPINE in
  !
  neighbor 10.0.0.2
    route-policy MARK_SPINE in
  !

# Outbound policy to prefer primary, install backup on loss
route-policy SELECT_PRIMARY
  if community matches-any PRIMARY_SPINE then
    set local-preference 200
  elseif community matches-any BACKUP_SPINE then
    set local-preference 100
  else
    set local-preference 50
  endif
  pass
end-policy

router bgp 65000
  address-family ipv4 unicast
    route-policy SELECT_PRIMARY in
  !

Explanation: Primary spine routes receive a higher local‑preference, ensuring selection while the next‑hop is reachable. If the primary spine becomes unreachable, the BGP session drops, primary routes are withdrawn, and backup routes (lower local‑preference) become the best path.

CLI Commands for Verifying Backup Intent

# Check which community is attached to a prefix
show route 10.10.10.0/24 detail
# Expected: Communities: 65000:100 (primary) or 65000:200 (backup)

# Confirm local‑preference used in selection
show bgp 10.10.10.0/24
# Expected: Localpref: 200 (primary) or 100 (backup)

# Simulate primary spine loss and observe switchover
test neighbor down 10.0.0.1
show bgp 10.10.10.0/24   # should now show backup community and lower localpref

The presence of the backup community and the change in local‑preference constitute an observable checkpoint that backup intent is active.

Implementing Stable Recursion

Understanding Recursion in Fabric Design

A leaf learns a BGP route with next‑hop = spine loopback (e.g., 10.0.0.1). The forwarding plane must resolve that loopback to a physical leaf‑spine link via the IGP. If the IGP reports multiple equal‑cost paths to the loopback, the leaf may install an ECMP set of next‑hops for the BGP route. Stable recursion requires that the set of IGP paths used for recursion does not fluctuate when the IGP metric to the loopback changes but the loopback remains reachable.

Configuring Stable Recursion for Fault Tolerance

Decouple the BGP next‑hop from the IGP‑resolved address by using a static or connected next‑hop reachable via a single, non‑ECMP IGP path.

• Option 1: Configure each spine’s loopback as a /32 with a single IGP adjacency (dedicated loopback‑only area or suppress ECMP on the loopback prefix).
• Option 2: Use BGP next‑hop-self on the spine so the leaf sees the spine’s physical interface address as the next‑hop, resolved via a single IGP path (the direct link).

Example Configuration (Juniper Junos)

# On each spine
set protocols bgp group LEAF type internal
set protocols bgp group LEAF neighbor <leaf-address> family inet unicast
set protocols bgp group LEAF neighbor <leaf-address> local-address <spine-loopback>
set protocols bgp group LEAF neighbor <leaf-address> next-hop-self   # leaf sees spine's physical IP

# Suppress ECMP on the loopback prefix in IS‑IS (or OSPF)
set protocols isis interface lo0.0 level 2 disable
set protocols isis interface lo0.0 level 2 metric 0   # still advertised but no ECMP

Explanation: The leaf’s BGP next‑hop becomes the spine’s physical interface IP (e.g., 10.1.0.5). The IGP only has one path to that IP—the direct leaf‑spine link—so recursion is stable: changes in IGP metrics elsewhere do not affect the resolved next‑hop.

Troubleshooting Common Recursion Issues

Designing Observable Failover Checkpoints

Importance of Observability in Failover

Without explicit checkpoints, operators infer failover from traffic patterns or latency spikes, delaying detection and complicating automation. Observable checkpoints enable:

• Automated validation – scripts query the RIB/FIB and assert that the backup community appears when the primary is withdrawn.
• Change‑control confidence – pre‑maintenance checkpoint tests confirm the backup path will be used.
• Post‑mortem clarity – logs show exactly when the backup route was installed, correlating with spine loss events.

Implementing Checkpoints for Real‑Time Monitoring

Export a telemetry stream carrying selected BGP path attributes (local‑preference, community, next‑hop) for prefixes of interest. Below is a vendor‑agnostic gNMI subscription example.

# Configure a gNMI target (OpenConfig example)
set system services gnmi server port 50051
set system services gnmi subscription SUB_FAILOVER
    path "/bgp-information/rib/ipv4-unicast/prefix[address='10.10.10.0/24']/best-path"
    mode sample
    sample-interval 1000   # ms
    encoding json_ipt

The telemetry payload includes:

{
  "prefix": "10.10.10.0/24",
  "best-path": {
    "next-hop": "10.1.0.5",
    "local-preference": 100,
    "communities": ["65000:200"]
  }
}

A monitoring system can raise an alert when local-preference drops from 200 to 100 and the community changes to 65000:200, indicating backup activation.

Code Example for Failover Checkpoint Evaluation (Python + gNMI)

import grpc
from gnmi import gnmi_pb2, gnmi_pb2_grpc
import json

def check_backup(stub):
    sub = gnmi_pb2.SubscriptionList(
        subscription=[
            gnmi_pb2.Subscription(
                path=gnmi_pb2.Path(
                    elem=[
                        gnmi_pb2.PathElem(name="bgp-information"),
                        gnmi_pb2.PathElem(name="rib"),
                        gnmi_pb2.PathElem(name="ipv4-unicast"),
                        gnmi_pb2.PathElem(name="prefix", key={"address": "10.10.10.0/24"}),
                        gnmi_pb2.PathElem(name="best-path")
                    ]
                ),
                mode=gnmi_pb2.SubscriptionMode.SAMPLE,
                sample_interval=10000000000  # 10 s in nanoseconds
            )
        ],
        mode=gnmi_pb2.SubscriptionMode.STREAM
    )
    request = gnmi_pb2.GetRequest(subscription=sub, encoding=gnmi_pb2.Encoding.JSON_IETF)
    for resp in stub.Subscribe(request, timeout=10):
        data = json.loads(resp.update[0].val.val)
        lp = data["best-path"]["local-preference"]
        comm = data["best-path"]["communities"]
        if lp == 100 and "65000:200" in comm:
            print("Backup path active")
        else:
            print("Primary path active")

This snippet subscribes to the BGP best‑path for the prefix, evaluates the local‑preference and community, and reports whether the backup is active.

By applying explicit backup intent, stable recursion, and observable failover checkpoints, the fabric transitions from relying on accidental symmetry to a deterministic, policy‑driven design that survives real failures with clear, operator‑visible signals.