Design Memo: Quorum Placement, Lease Timeouts, Fencing, and Operator Override for WAN‑Resilient Controllers

Author: Sophia Lin – Cloud & Automation Architect
Date: 2025‑09‑26

Introduction

Controllers that orchestrate network services (SD‑WAN overlays, service‑mesh control planes, NFV orchestrators) must stay available when a WAN partition isolates one or more regions. The two failure modes to avoid are:

Global freeze – the controller cannot form a quorum and stops processing traffic.
Dual writer (split‑brain) – disjoint partitions each believe they have a valid quorum and issue conflicting updates.

Quorum placement and lease timeouts are the primary levers for influencing how quickly a partition detects loss of majority and whether it may continue operating. The design must balance fault‑detection speed against availability under transient WAN degradation.

Quorum Placement Strategies

Strategy	Description	Typical Use‑Case	Fault‑Tolerance Model
Static odd‑sized voter set	Fixed number of voting nodes (e.g., 3, 5, 7) placed a priori.	Small‑to‑medium clusters with predictable low latency (< 50 ms RTT).	Tolerates ⌊(N‑1)/2⌋ node failures.
Weighted voting	Each node carries a weight; quorum is reached when sum of weights ≥ threshold.	Heterogeneous sites (large hub weight = 2, spokes weight = 1).	Tolerates failures up to weight‑threshold‑1.
Witness‑only nodes	Non‑voting nodes that participate in lease/heartbeat exchanges but do not count toward quorum.	Geo‑distributed deployments where adding a full voter per site is costly.	Provides tie‑breaking without increasing failure tolerance.
Flexible quorums (Flexible Paxos / Flexible Raft)	Read quorum (R) and write quorum (W) can differ; only need R+W > N.	Read‑heavy workloads; allows smaller write quorum for larger read quorum.	Tunable trade‑off between write latency and fault tolerance.
Hierarchical / regional quorum	Each region runs a local quorum; a global coordinator needs only a subset of regional leaders to agree.	Multi‑region services with low intra‑region but high inter‑region latency.	Failure tolerance is per‑region; global progress requires at least one healthy region plus enough regional leaders.

Placement Constraints

Voters must reside in independent failure domains (different power, network, and geographic zones).
Maximum one‑way latency between any two voters must be bounded and known; lease timeouts are set relative to this bound.
Adding voters increases message complexity (O(N²) heartbeats) and commit latency (linear in the slowest voter).

Lease Timeout Configuration

A lease is a time‑bounded promise that a leader holds the right to propose updates. If the leader fails to renew its lease before expiration, followers may initiate a new election.

Lease duration (T_lease) – typically a multiple of the heartbeat interval.
Heartbeat interval (T_hb) – frequency at which the leader sends lease renewal messages.
Election timeout (T_election) – minimum time a follower waits without hearing from a leader before starting an election (often randomized).

Configuration Guidelines (based on measured RTT, r, between voter sites)

Lower bound – T_lease > 2 × max_one_way_latency to avoid premature expiration from transient loss.
Example: max RTT = 80 ms (one‑way ≈ 40 ms) → T_lease ≥ 160 ms.
Upper bound – T_lease should be < MTBF of the network link to ensure timely detection of actual partitions.
Heartbeat vs. lease – T_hb = T_lease / k, where k ∈ [2, 5]. Smaller k yields faster detection but higher overhead.
Randomized election timeout – T_election = T_lease × [0.5, 1.5] (or similar) to reduce collision probability.

Trade‑offs

Parameter	Smaller Value	Larger Value
T_lease / T_hb	Faster leader failure detection → quicker failover, but higher false‑positive rate under jitter.	Slower detection → fewer false positives, but longer unavailability during a real partition.
Election timeout spread	Narrow spread → higher probability of simultaneous elections (split‑vote).	Wide spread → lower collision probability but longer worst‑case election time.
Weighted voting	Allows small sites influence without adding many voters.	Increases complexity of weight‑change procedures; mis‑weighted configs can unintentionally create minority quorums.

Fencing Mechanisms

Even with correct quorum and lease settings, a partitioned leader may continue to believe it is valid while followers have elected a new leader. Fencing ensures the old leader cannot cause divergent state.

Mechanism	How It Works	Typical Deployment	Pros	Cons / Failure Modes
STONITH	Power‑cycle or reset the suspect node via out‑of‑band management (IPMI, iLO, iDRAC, etc.).	Physical servers, bare‑metal controllers.	Guarantees node is powered off; simple to audit.	Requires reliable out‑of‑band network; mis‑fire causes service disruption.
I/O Fencing (SCSI‑3 Persistent Reservations)	Reserve access to shared storage; only the node holding the reservation may perform I/O.	Shared‑disk clusters (Oracle RAC, some NFV VIMs).	Prevents split‑brain at storage layer.	Requires SCSI‑3 capable storage; not applicable to pure state‑machine controllers (etcd, Consul).
Network‑Based Fencing	Disable the node’s network interface (e.g., via API to switch, or null‑route) or revoke its TLS certificates.	Virtualized or cloud controllers where NIC/API access is available.	No power cycling; can be granular (specific services).	Dependent on control‑plane network; if the control plane is partitioned, fencing may fail.
Lease‑Based Fencing	Leader must hold a lease (e.g., from a lock service) to write; loss of lease automatically revokes write authority.	Etcd, Consul, ZooKeeper (via session leases).	Ties fencing directly to consensus lease; no extra hardware.	If lease service itself is partitioned, both sides may think they hold the lease—requires careful placement of lease service voters.
Quorum‑Based Fencing (Witness Node)	A designated witness (non‑voting) must acknowledge any write; loss of witness contact blocks writes.	Custom controller designs; rarely used in off‑the‑shelf stacks.	Simple to implement; witness can be placed in a third site for tie‑breaking.	Adds latency; witness becomes a single point of failure unless replicated.

Choosing a Fencing Method

State‑machine controllers (etcd, Consul, ZooKeeper) – built‑in lease mechanism is usually sufficient, provided the lease service’s voter set matches the controller’s fault‑tolerance goals.
Bare‑metal or hybrid environments – STONITH remains a robust fallback when the controller runs on VMs that can be power‑cycled.
Public‑cloud settings – network‑based fencing (API calls to revoke security groups or IAM roles) is fast and auditable.

Operator Override Paths

Automated fencing may be undesirable during planned maintenance or when an operator wishes to force a specific node to stay leader (e.g., to avoid a costly re‑sync). Override paths must be:

Explicit – require a deliberate command, not a side‑effect of a configuration change.
Auditable – logged with timestamp, operator identity, and reason.
Reversible – allow the system to return to automatic mode after the override ends.
Isolated – cannot bypass core safety invariants (e.g., must still respect lease expiration unless the operator also updates the lease timeout).

Common Patterns

Override Mechanism	Implementation Example	Safety Guardrails
Manual leader transfer	`etcdctl member promote <node-id>` or `consul operator raft remove-peer` followed by `add-peer`	Only allowed if current leader is healthy; requires lease renewal from the target node.
Force quorum reset	`etcdctl snapshot restore <snapshot>` then `--initial-cluster` with a new voter set	Requires a valid snapshot; system refuses if snapshot term is older than current term.
Disable automatic fencing	`crm configure property stonith-enabled=false` (Pacemaker)	Must be accompanied by a maintenance‑window flag; monitoring alerts on disabled fencing.
Lease timeout override	Update etcd `--election-timeout` flag via rolling restart or dynamic reconfiguration (if supported)	Change is logged; monitoring verifies new timeout > 2×RTT; rollback restores original value.
Emergency read‑only mode	Set controller to reject write requests (e.g., via feature flag or policy) while allowing reads	Guarantees no divergent writes; used when network partition is suspected but not confirmed.

All override actions should go through an approval workflow (e.g., ChatOps with required sign‑off, or a ticket‑based change management system) to satisfy the “operator override path” requirement.

Designing for WAN Partition Survival

The goal is to allow each region to continue operating as long as it can maintain a local quorum, while preventing global freeze or dual writers when the WAN partitions. The design combines quorum placement, lease tuning, fencing, and explicit operator gates.

Regional Fault Isolation Techniques

Regional Voter Subsets – Assign a minimum odd number of voters per region (e.g., 3 voters in each of three regions). Inter‑region communication is only needed for global commit decisions; local reads/writes can proceed with the regional quorum if the application permits eventual consistency or read‑your‑writes semantics.
Witness‑Only Tie‑Breaker – Place a lightweight witness node in a third, highly‑available location (e.g., a cloud region with excellent connectivity). The witness does not store state but participates in lease heartbeats to break ties when two regions each have half the voters.
Asynchronous State Shipping – For operations requiring strong consistency across regions (e.g., global policy updates), use an asynchronous replication stream that is not on the critical path; the local controller can acknowledge the write locally and later propagate the update.
Dynamic Voter Re‑configuration – When a region detects a sustained WAN loss (> threshold), it can temporarily demote its voters to non‑voting observers and rely on its local quorum for all decisions. The change is reversed automatically when connectivity heals.

Global Freeze Prevention Strategies

Lease Timeouts Greater Than WAN RTT – Set T_lease to comfortably exceed the worst‑case observed WAN round‑trip time (including jitter). This prevents the leader from expiring its lease merely because a packet is delayed.
Pre‑Vote / Vote‑Request Optimization – In Raft, a candidate first sends a PreVote to gauge support; if it does not receive a majority, it aborts the election, avoiding unnecessary term increments that could prolong unavailability.
Read‑Only Lease Renewal – Allow followers to renew the leader’s lease passively (by forwarding heartbeats) when they suspect a transient partition, reducing the chance of spurious elections.
Staggered Election Timeouts – Randomize election timeouts per node with a wide enough spread (e.g., ±40 % of base timeout) to reduce the probability that multiple regions start elections simultaneously.

Dual Writer Event Mitigation

Fencing via Lease Expiration – Ensure that any node that loses contact with the lease service cannot renew its lease; once the lease expires, the node must step down and refuse to accept client writes. The lease service itself must be quorum‑protected with the same voter placement rules.
Split‑Brain Detector (SBD) – Some stacks (e.g., Pacemaker) implement an SBD that watches for simultaneous leadership claims and triggers fencing on both sides if detected.
Write‑Barrier on Partition Heal – When the WAN link restores, the controller executes a barrier operation: all nodes pause writes, exchange their latest committed terms, and only the node with the highest term is allowed to continue as leader. This guarantees convergence.
Operator‑Gated Write Enable – After a partition heals, an automated script may require operator confirmation before re‑enabling writes across regions, providing a final human check against accidental dual writer scenarios.

Troubleshooting Quorum and Lease Timeout Issues

Effective troubleshooting relies on observable metrics, logs, and a small set of CLI commands. The following sections assume a typical etcd‑based controller; analogous commands exist for Consul (consul operator raft) and ZooKeeper (zkCli.sh).

Identifying and Resolving Quorum Loss

Symptom	Diagnostic Command	Interpretation
No leader, cluster unavailable	`etcdctl endpoint status --cluster -w table`	Shows which nodes believe they are leader; blank leader field indicates no quorum.
Member listed as “unhealthy”	`etcdctl member list`	Look for `false` under `isLearner` or `isLeader`; check `unhealthy` flag.
Election logs flooding	`journalctl -u etcd` (or container logs)	Repeated “failed to send heartbeat” or “election timeout” messages indicate lease/heartbeat issues.
Split‑brain suspicion (two leaders)	`etcdctl endpoint status` from multiple clients; compare `leader` fields.	Two different leaders → potential dual writer.

Resolution steps

Verify network connectivity between voters (ping, traceroute, tcptrace).
Check lease renewal latency and heartbeat response times via etcdctl endpoint status or Prometheus metrics (etcd_server_leader_changes_seen_total, etcd_network_client_grpc_received_bytes_total).
If a node is consistently unhealthy, inspect its logs for lease renewal failures and consider replacing or isolating the node.
In case of split‑brain, identify the partition with the higher term (via etcdctl endpoint status → leader and term) and fence the lower‑term leader using the chosen fencing mechanism.
After fencing, restart the isolated node and let it rejoin the cluster; verify that a single leader is elected and that the cluster health returns to normal.

End of memo.

Quorum design for multi-region network controllers