VIP Failover Without Session Ownership

Introduction to Stateful Load Balancers and Outage Scenarios

Overview of Stateful Load Balancing

Stateful load balancers (SLBs) maintain per‑connection or per‑client state so that subsequent packets from the same flow are directed to the same backend server. Typical mechanisms include:

• Source‑IP affinity – hash of client IP → backend.
• Cookie‑based persistence – LB inserts or reads a cookie (e.g., JSESSIONID, BIGipServer~pool~server).
• SSL session ID reuse – for TLS termination.
• Stick‑tables (HAProxy) or persist profiles (F5 BIG‑IP‑ADC) that store a mapping of client identifier → backend for a configurable timeout.

The state is usually kept in memory on the LB node; in an HA pair it must be replicated to the standby node so that a failover does not break existing flows.

Importance of Session Persistence

Many applications rely on sticky sessions: shopping carts, authentication tokens, in‑process caches, or any server‑side state that is not shared across backend nodes. If persistence is lost after a failover, a client that had an established session will be sent to a different backend on the next request, causing:

• Loss of transient state (e.g., cart contents).
• Re‑authentication prompts.
• Application errors due to missing session data.

From the network perspective the VIP remains reachable, health checks to the backend pool continue to succeed, and routing tables show a valid path—yet the user experience degrades. This mismatch creates three divergent narratives: routing success, health‑check green, and user‑visible breakage.

Reconstructing the Outage

Initial Failover and VIP Reachability

At 02:13 UTC a core switch port flapped, triggering VRRP failover of the active‑passive LB pair (lb01 → lb02). The virtual IP (10.0.0.100/24) remained owned by lb02; ARP replies and ICMP echo replies continued without interruption.

# Before failover
$ ping -c 3 10.0.0.100
64 bytes from 10.0.0.100: icmp_seq=1 ttl=64 time=0.3 ms
64 bytes from 10.0.0.100: icmp_seq=2 ttl=64 time=0.3 ms
64 bytes from 10.0.0.100: icmp_seq=3 ttl=64 time=0.3 ms

# Immediately after VRRP transition (no packet loss)
$ ping -c 3 10.0.0.100
64 bytes from 4.0.0.100: icmp_seq=1 ttl=10.0.0.100: icmp_seq=1 ttl=64 time=0.3 ms
64 bytes from 10.0.0.100: icmp_seq=2 ttl=64 time=0.3 ms
64 bytes from 10.0.0.100: icmp_seq=3 ttl=64 time=0.3 ms

Loss of Session Ownership Loss of Session Ownership and Its Implications

The LB pair used VRRP only for HA; state synchronization was disabled (a common oversight when migrating from a legacy active‑active setup). Consequently, the stick‑table on lb01 was not replicated to lb02. When lb02 took over the VIP, its stick‑table was empty. New packets belonging to existing flows were load‑balanced using the default algorithm (round‑robin) instead of being pinned to the original backend.

• Routing success – the VIP answered ARP/ICMP, and TCP SYNs arrived at lb02.
• Health checks – the LB’s health‑check probes (HTTP GET /health) continued to succeed because they were new connections and the backend pool was healthy.
• User‑visible breakage – a user whose session was tied to backend web03 via a persistence cookie now hit web01 on the next request, causing the application to treat the request as unauthenticated and redirect to login.

Technical Details of the Outage

The root cause was the absence of stick‑table replication (peers section missing or disabled) in the HAProxy configuration.

Troubleshooting the Outage

Identifying Symptoms and Gathering Logs

1. User‑facing metrics – spike in**02/401, increase in average response time.
2. LB logs – repeated stick-table lookup miss messages.
3. Network traces – TCP SYN‑ACK from VIP to varying backend IPs for the same client IP.
4. VRRP logs – clean master transition, no split‑brain.

Log collection commands (example on Linux‑based HAProxy):

# Collect recent HAProxy logs
$ sudo journalctl -u haproxy --since "2025-09-24 02:10:00" > haproxy_before_failover.log

# Capture VRRP state changes
$ sudo grep VRRP /var/log/syslog | grep -E "2025-09-24 02:1[0-9]" > vrrp_transitions.log

# Dump stick-table at intervals (requires stats socket)
$ echo "show table http-stick" | sudo socat stdio /var/run/haproxy.sock > sticktable_02_13_00.txt
$ echo "show table http-stick" | sudo socat stdio /var/run/haproxy.sock > sticktable_02_13_05.txt

Analyzing Network Traffic and Load Balancer Configurations

• tcpdump on the LB interface to see which backend MAC/IP answered SYN‑ACK:

$ sudo tcpdump -i eth0 -nn -s 0 -w /tmp/lb_failover.pcap 'dst port 80 or dst port 443' &
# After a few minutes, stop capture and examine:
$ sudo tshark -r /tmp/lb_failover.pcap -Y "tcp.flags.syn==1 && ip.src==10.0.0.100" -T fields -e ip.dst | sort | uniq -c

Output showed a roughly equal distribution across web01–web04, confirming loss of persistence.

• Configuration diff – compare active and standby configs:

$ sudo diff -u /etc/haproxy/haproxy.cfg.lb01 /etc/haproxy/haproxy.cfg.lb02
--- haproxy.cfg.lb01 2025-09-20 12:00:00
+++ haproxy.cfg.lb02 2025-09-20 12:00:00
@@
-    peers haproxy_peers
-        peer lb01 10.0.0.10:1024
-        peer lb02 10.0.0.11:1024
+    # peers section disabled for testing

The peers block was commented out on lb02, meaning no state sync.

Code Examples for Debugging Load Balancer Issues

CLI Commands for Load Balancer Inspection

Scripting for Automated Troubleshooting

A simple Bash script that polls the stick‑table size and alerts if it drops below a threshold (indicating loss of replication):

#!/usr/bin/env bash
SOCKET="/var/run/haproxy.sock"
THRESHOLD=1000   # expect at least 1k entries under normal load
INTERVAL=15      # seconds

while true; do
    SIZE=$(echo "show table http-stick" | socat stdio "$SOCKET" | awk '/^Table:/ {print $2}')
    if [[ -z "$SIZE" || "$SIZE" -lt "$THRESHOLD" ]]; then
        logger -t haproxy-stick "ALERT: stick-table size $SIZE (< $THRESHOLD)"
        # optional: trigger pagerduty/webhook
    fi
    sleep $INTERVAL
done

Run as a systemd service to get early‑warning continuous visibility.

Routing Success and Health Checks

Understanding Routing Success in the Context of the Outage

• Layer‑2/Layer‑3 reachability – The VIP responded to ARP and ICMP because VRRP only governs MAC ownership; the underlying IP stack on the LB remained up.
• TCP SYN acceptance – The LB’s listener (bind 10.0.0.100:80) accepted new SYNs regardless of stick‑table state; the SYN‑ACK was sent after the LB selected a backend via its load‑balancing algorithm.
• Result – From a network‑engineer’s viewpoint, the service was “up”: no packet loss, latency unchanged, and routing tables showed a valid next‑hop.

Health Check Configurations and Their Role in the Outage

Health checks were configured as layer‑7 HTTP GET to /health on each backend, performed every 5 seconds with a rise of 2 and fall of 3. Because each check established a new TCP connection, the LB treated them as independent flows and selected a backend based on the current algorithm (round‑robin). The backend pool was healthy, so all checks passed, giving a false sense of normality.

HAProxy health‑check snippet (present on both nodes):

backend web_pool
    mode http
    balance roundrobin          # <-- used for health checks when stick-table miss
    option httpchk GET /health
    http-check expect status 200
    server web01 10.0.1.10:80 check inter 5s rise 2 fall 3
    server web02 10.0.1.11:80 check inter 5s rise 2 fall 3
    server web03 10.0.1.12:80 check inter 5s rise 2 fall 3
    server web04 10.0.1.13:80 check inter 5s rise 2 fall 3

If the stick-table had been populated, the health‑check traffic would still have been load‑balanced, but the user traffic would have been persisted. The key point: health‑check success does not guarantee session persistence.

Example Code for Configuring Health Checks