Fault-Injecting a Lost PodCIDR Advertisement

Introduction to Steady-State Hypothesis

Definition of Steady-State Hypothesis

A steady‑state hypothesis is a formal statement about the expected behavior of a system when no external perturbations are applied. In networking, it describes the stable equilibrium of control‑plane state (RIB, FIB, adjacency tables) and data‑plane forwarding under normal traffic loads. The hypothesis includes:

• All nodes advertise reachability for their locally hosted pod IP prefixes.
• Each node’s forwarding table contains a valid next‑hop for every pod prefix.
• End‑to‑end packet loss between healthy pods is zero (within measurement noise).
• Convergence timers have expired; no route flaps are pending.

Importance of Steady-State Hypothesis in Network Stability

Establishing a steady‑state baseline lets an operator isolate the effect of a deliberate fault injection. By comparing the observed post‑fault state to the hypothesis, one can determine whether:

• The failure is contained to the suppressed advertisement (only the affected node’s pods lose reachability).
• The control plane reroutes traffic via alternative paths (indicating resilience mechanisms such as multipath or encapsulation).
• The fault propagates, pulling healthy nodes into a blast radius (e.g., blackholing, routing loops, or excessive reconvergence).

The hypothesis also defines the stop condition for the experiment: when the observed metrics (packet loss, latency, route churn) return to the values predicted by the hypothesis, the system has re‑stabilized.

Understanding CNI and Pod Reachability

Overview of Container Network Interface (CNI)

The Container Network Interface (CNI) is a specification and library suite that configures network namespaces for containers. A CNI plugin is invoked by the container runtime (e.g., containerd, cri‑o) with a JSON network configuration and is responsible for:

1. Allocating an IP address from an IPAM plugin.
2. Creating the necessary virtual interfaces (veth pair, bridge, macvlan, etc.).
3. Programming the host’s forwarding tables (routes, ARP/NDP entries, firewall rules) so that the pod’s IP is reachable from other nodes.
4. Cleaning up resources on pod deletion.

CNI does not prescribe a particular control‑plane protocol; the method by which pod reachability is advertised depends on the plugin (e.g., host‑local routes, BGP, VXLAN encapsulation, or SRv6).

Role of CNI in Pod Reachability and Network Stability

In a Kubernetes cluster, pod reachability is typically expressed as a route to the pod’s IP (or to the node’s pod CIDR) in each node’s routing table. The CNI plugin ensures that:

• The route is present as long as the pod exists.
• The route is withdrawn when the pod terminates or the node becomes unhealthy.
• The route is consistent across nodes, preventing black holes or duplicate address issues.

If the CNI fails to program or withdraw a route correctly, the control plane diverges from the forwarding plane, leading to packet loss, suboptimal paths, or forwarding loops.

CNI Plugins and Their Impact on Network Behavior

Suppressing Node Pod Reachability Advertisement

Methods for Suppressing Pod Reachability Advertisement

Suppression can be achieved at different layers of the CNI stack:

1. Remove the host route for the pod CIDR/IP (ip route del …).
2. Disable the CNI plugin’s agent (e.g., stop calico-node, flanneld, or cilium-agent).
3. Withdraw the route via the plugin’s control plane (e.g., calicoctl node status --withdraw, flannel subnet update).
4. Block traffic with firewall rules (iptables -A OUTPUT -d <pod‑cidr> -j DROP).
5. Misconfigure IPAM so the plugin believes the node has no allocatable IPs, causing it to withdraw existing routes.

For a controlled experiment, the most repeatable method is to programmatically delete the host route that the CNI installed, leaving the CNI agent running but unable to forward traffic for the affected prefixes.

Tools and Techniques for Simulating Network Failures

• ip route – add/delete/modify kernel routes.
• iptables/nftables – drop or redirect packets to simulate blackholing.
• tc – add delay, loss, or corruption via netem to emulate intermittent reachability.
• CNI‑specific CLI – calicoctl, fluctl, cilium-cli to trigger route withdrawals or agent restarts.
• Chaos‑mesh / LitmusChaos – Kubernetes‑native chaos injection that can target CNI pods or network namespaces.

Example Code for Suppressing Pod Reachability using CLI

#!/usr/bin/env bash
set -euo pipefail

NODE_NAME="worker-3"
POD_CIDR="10.244.3.0/24"   # adjust to match your cluster's CIDR allocation

# Identify the node's internal IP (used for SSH or direct exec)
NODE_IP=$(kubectl get node "$NODE_NAME" -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}')

# SSH into the node and delete the host route for the pod CIDR
ssh "$NODE_IP" "sudo ip route del $POD_CIDR"

echo "Suppressed pod reachability advertisement for $NODE_NAME ($POD_CIDR)"

If the cluster uses Flannel, replace the ip route del with sudo ip link set dev flannel.1 down to bring the VXLAN interface down, which implicitly withdraws the learned remote VTEP entries.
For Cilium, you could run cilium status --delete-endpoints <label> to force endpoint removal, which causes the agent to withdraw the associated L3 routes.

Measuring Network Behavior and CNI Response

Monitoring Network Traffic and Pod Reachability

After suppression, the operator should verify:

• Control‑plane state – routes present in each node’s FIB (ip route show).
• Data‑plane loss – packet loss or latency increase when probing the suppressed node’s pods.
• Control‑plane churn – frequency of route updates (netstat -s or ss -s).
• CNI agent health – logs and metrics indicating whether the agent detected the missing route.

Analyzing CNI Logs and Network Metrics

• Calico – /var/log/calico/felix.log contains lines like info] Removing route 10.244.3.0/24 via dev eth0.
• Flannel – /var/log/flannel.log shows Failed to submit subnet lease or Lost lease for subnet.
• Cilium – cilium monitor output drops L3 packets; cilium status shows Endpoints: 0 for the affected node.
• Generic – journalctl -u cni* (if a wrapper service exists) or kubelet logs (/var/log/kubelet.log) for CNI invocation errors.

Key metrics to collect:

Example CLI Commands for Monitoring Network Behavior

# 1. Verify route suppression on the target node
ssh worker-3 "ip route show | grep 10.244.3.0/24"
# Expected: no output (route removed)

# 2. Test reachability from a pod on another node
kubectl run -i --tty --rm debug --image=alpine --restart=Never -- \
  sh -c "apk add --no-cache iputils && \
         for i in {1..10}; do \
           ping -c 1 10.244.3.42 && echo 'OK' || echo 'LOSS'; \
         done"

# 3. Collect Felix logs (Calico)
ssh worker-3 "journalctl -u calico-felix -f | grep -i route"

# 4. Measure inter‑node RTT to a healthy pod for comparison
kubectl exec -it debug -- ping -c 5 10.244.1.10   # pod on worker-1

# 5. Export Prometheus metrics (if CNI exposes them)
curl http://worker-3:9090/metrics | grep