Zone drains that outpace route withdrawal propagation

Introduction to Node Drain Workflows

Node drain workflows prepare a compute node for maintenance or decommissioning by evacuating workloads. In Kubernetes‑centric environments the typical steps are:

1. Cordon – kubectl cordon <node> marks the node unschedulable.
2. Evict pods – kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --force evicts all pods.
3. CNI route withdrawal – The CNI plugin withdraws host‑specific routes or service‑IP advertisements from the underlay/overlay fabric.
4. Confirm withdrawal – After the CNI acknowledges the withdrawal, the node can be power‑cycled or replaced.

Correctness depends on timely propagation of those withdrawals through the routing control plane so forwarding decisions no longer point to the draining node.

Why Route Withdrawals and Fabric Policy Matter

A BGP withdrawal tells peers a prefix is no longer reachable. If delayed or lost, stale routes linger in upstream RIB/FIB, causing black‑holing or mis‑directed traffic.

Fabric policy (ACLs, route‑maps, VRF‑local policy) enforces intent such as “traffic to service X must egress via zone A”. When a route withdrawal updates the RIB, the policy is re‑evaluated; if the withdrawal hasn’t arrived, the policy may still forward packets toward the stale next‑hop, breaking zone isolation and creating a painful window where cross‑zone traffic hits a node that no longer hosts the service.

Route Reflectors and Fabric Policy

Role of Route Reflectors

In large IP‑fabric designs a full‑mesh iBGP peer set does not scale. Route reflectors (RRs) let clients peer only with the RR, which re‑advertises received routes according to iBGP loop‑prevention rules:

• Routes learned from a client are re‑advertised to all other clients and non‑clients.
• Routes learned from a non‑client are re‑advertised only to clients.
• The RR adds its own cluster‑list attribute to prevent re‑advertisement back to the originator.

The RR does not alter attributes like LOCAL_PREF, MED, or AS_PATH unless explicitly configured; it may change NEXT_HOP to its own address if next-hop-self is set, otherwise it preserves the original next‑hop.

Fabric Policy Functionality

Applied at leaf switch ingress/egress (or VRF level on spines), fabric policy typically includes:

• Match criteria – prefix lists, community lists, or application‑specific tags (e.g., Kubernetes namespace labels encoded as BGP communities).
• Actions – permit/deny, set LOCAL_PREF, MED, or modify next‑hop via set ip next-hop.
• Evaluation order – route‑map sequences or ACL chains are processed sequentially; the first match decides forwarding.

When a route is installed in the FIB, the fabric policy determines whether the packet is forwarded, dropped, or redirected to another zone. Correct forwarding therefore depends on both a valid RIB entry and the policy’s evaluation of that entry.

Interaction Between RRs and Fabric Policy

1. Control‑plane stage – BGP updates (advertisements or withdrawals) flow: CNI agent → leaf switch (BGP speaker) → route reflector → other leaf switches. The RR respects client/non‑client rules and propagates the withdrawal throughout the fabric.
2. Policy stage – Each leaf switch updates its local RIB/FIB, then applies its fabric policy to determine the final forwarding action for matching traffic.

If the withdrawal stalls at the RR stage (due to processing backlog, MRAI throttling, or a stuck peer session), leaf switches retain the stale route. The fabric policy continues to match that prefix and forwards traffic toward the withdrawn next‑hop, violating zone isolation.

Node Drain Workflows and Route Withdrawals

Normal Operation

Under nominal conditions the sequence is:

1. Cordon – kubectl cordon <node> marks the node unschedulable.
2. Pod termination – Kubelet sends SIGTERM; containers exit, pod moves to Terminating.
3. CNI withdrawal – CNI detects pod deletion and issues a BGP WITHDRAW for the host route (typically /32 of the pod IP or node’s service‑IP /32) to the leaf switch’s BGP peer.
4. RR propagation – Leaf (as RR client) forwards the withdrawal to its configured RR(s). The RR processes the UPDATE, updates its BGP table, and after the MinRouteAdvertisementInterval (MRAI, default 5 s for iBGP) re‑advertises the withdrawal to all other clients.
5. Leaf RIB/FIB update – Each leaf receives the withdrawal, removes the prefix from its BGP RIB, triggers a FIB update, and re‑applies fabric policy. Traffic for the withdrawn prefix is now dropped or redirected per policy.
6. Drain completion – Once the CNI confirms all withdrawals have been acknowledged (or after a configurable timeout), the node is considered drained and can be power‑cycled.

Impact of Rapid Pod Removal

If pods are removed faster than the BGP withdrawal pipeline can drain, several failure modes appear:

• Withdrawal burst – CNI may generate dozens of WITHDRAW updates in a sub‑second window. Per‑peer MRAI throttling can queue or drop updates if the send queue exceeds its limit.
• RR processing lag – The RR must run its decision process for each UPDATE. In high‑scale fabrics the RR’s BGP process can become CPU‑bound, increasing update‑processing latency (observable as increased BGP update queue depth).
• Stale next‑hop persistence – While the RR processes the burst, leaf switches continue to advertise the original route. Traffic arriving during this window uses the stale next‑hop, pointing to a node already being drained.
• Policy mis‑application – Fabric policy may still match the stale prefix and forward packets toward the drained node. A set ip next-hop action can exacerbate the misdirection.

Observable symptoms: temporary latency increase or packet loss for flows to the drained service, spikes in BGP updates received on the RR, and a rise in out-discards on the leaf facing the draining node.

Scenarios Where Traffic Continues to Cross Zones

Consider a three‑zone fabric (Zone A, B, C) where a service runs only in Zone A. Fabric policy: “traffic to service‑X prefix must be forwarded only within Zone A; any attempt to exit Zone A is dropped”.

In each case the root cause is a temporal mismatch between the data‑plane pod removal rate and the control‑plane withdrawal propagation latency.

Troubleshooting Node Drain Workflows

Identifying Symptoms

Operators should watch for:

• Increased retransmits or application timeouts for client‑to‑service connections originating outside the draining zone.
• Spikes in BGP updates received on the RR (show ip bgp summary → Received column) coinciding with drain start times.
• Growth of the BGP update queue (show bgp neighbors <peer> received-routes or vendor‑specific show bgp process detail).
• Persistent stale routes in the leaf RIB (show ip route <prefix> still showing the drained node as next‑hop after drain completion).
• Fabric policy hit counters showing matches on the stale prefix (show access-lists or show route-map counters increasing while the service is supposedly absent).
• Link‑level loss on the leaf‑to‑node interface (show interfaces counters errors) due to black‑holed traffic.

Tools and Techniques

• BGP monitoring – Use bgpwatch, Prometheus bgp_exporter, or vendor telemetry to graph bgpUpdatesIn, bgpUpdatesOut, and bgpPeerState. Alert on sudden increases in bgpUpdatesIn > 2× baseline during drain windows.
• CNI audit logs – Most CNIs log each route withdrawal/addition (calico-node, cilium-agent). Correlate timestamps with BGP update events.
• Flow sampling – Enable sFlow or IPFIX on leaf switches; filter on the service prefix and observe the egress interface. A temporary shift to the node’s interface after drain start indicates stale forwarding.
• Kubernetes events – kubectl get events --field-selector involvedObject.name=<node> shows pod terminations and when the node is marked Ready=False.
• Policy hit counters – On Cisco/Nexus: show access-lists includes match counts; on Juniper: show firewall; on Arista: show ip access-lists.

Debugging Route Withdrawal Propagation

1. Verify the withdrawal was sent – On the node, check CNI logs for WITHDRAW messages and confirm the TCP segment left the host:

   tcpdump -i any port 179 -w withdraw.pcap

2. Check peer state – On the leaf switch:

   show bgp neighbors <peer> state

   Should be Established; Idle/Active indicates the withdrawal never left the host.
3. Inspect RR queue –

   show bgp process detail          # Cisco
   show bgp summary                 # Juniper (look for Update queue depth)

   A non‑zero depth during drain indicates backlog.
4. Confirm RR re‑advertisement – On an RR client leaf:

   show bgp <prefix>

   The withdrawn route should appear as * (not present) after the MRAI interval. If still present, trace the path:

   show bgp ipv4 unicast <prefix>

   Look for the Originator ID equal to the RR’s own address – indicates the RR has not processed the withdrawal.
5. Validate fabric policy – After confirming the route is gone from the RIB:

   show route-map <map>

   Ensure the match counter for the prefix is zero. A non‑zero counter suggests the policy is still seeing the route (possible stale FIB entry).
6. Force a soft reset –

   clear ip bgp <peer> soft in

   Triggers a route‑refresh and withdraws any stale state without tearing down the session. Observe if the prefix disappears immediately.

If the prefix disappears after a soft reset but returns later, the issue is likely a mis‑behaving CNI re‑injecting the route (e.g., due to a stale endpoint object). If the prefix persists despite soft reset, the problem lies in the RR or fabric policy configuration.

Code and CLI Examples

Example Node Drain Workflow (bash)

# 1. Cordon the node
kubectl cordon worker-03

# 2. Drain pods (ignore DaemonSets, delete emptyDir data, force if needed)
kubectl drain worker-03 --ignore-daemonsets --delete-emptydir-data --force

# 3. Verify CNI has sent withdrawals (Calico example)
journalctl -u calico-node | grep "WITHDRAW"

# 4. Check BGP peer state on leaf switch (Cisco NX-OS)
show bgp neighbors 10.0.0.5 state

# 5. Monitor RR update queue (Juniper)
show bgp summary | match "Update queue"

# 6. After drain, confirm route removal
show ip route 10.96.0.0/12   # should not list worker-03 as next-hop

# 7. Power‑cycle or replace the node
# (e.g., via your infrastructure automation)

Sample Fabric Policy (Cisco IOS‑XR)

route-map SERVICE-X-POLICY permit 10
 match ip address prefix-list SERVICE-X-PREFIX
 set ip next-hop 10.0.0.1   ! egress via Zone A leaf
!
route-map SERVICE-X-POLICY deny 20
 match ip address prefix-list ANY
!
interface Bundle-Ether10
 ip address 10.0.0.2/30
 ip policy route-map SERVICE-X-POLICY

Sample CNI Withdrawal Log (Calico)

time="2025-09-24T14:32:07Z" level=info msg="Sending BGP WITHDRAW for pod IP 10.244.3.45/32"
time="2025-09-24T14:32:07Z" level=info msg="BGP UPDATE sent to peer 10.0.0.5 (AS 65000)"

These examples illustrate the commands and configurations you can use to observe, verify, and troubleshoot the interaction between node drain workflows, BGP withdrawals, route reflectors, and fabric policy. Adjust the syntax to match your specific vendor and CNI implementation.