Introduction to CNI and Pod Routing
Container Network Interface (CNI) is a plug‑in based specification for configuring container networks. CNI plugins are invoked by the container runtime (containerd, cri‑o, etc.) during pod creation and deletion. They fall into two broad categories that dictate how pod‑to‑pod traffic traverses the cluster:
| Category | Typical Plugins | Forwarding Model | Control‑Plane Mechanism |
|---|---|---|---|
| Overlay | Flannel (VXLAN), Weave Net, Calico (IP‑in‑IP), Canal (Flannel + Calico) | Encapsulates pod packets in an overlay header (VXLAN, Geneve, IP‑in‑IP) and forwards them via the underlying L2/L3 underlay network. | The overlay control plane (often a simple daemon or a distributed key‑value store) maintains VTEP‑to‑node mappings; no routing of pod CIDRs is required in the underlay. |
| Routed | Calico (BGP mode), Cilium (native routing), Kube‑Router, Romana | Pod IPs are advertised as host routes (typically /32) into the fabric; forwarding relies on standard IP routing (ECMP, BGP, OSPF, static). | A routing protocol (most commonly BGP) runs on each node or on a dedicated route‑reflector, distributing pod‑IP/host‑route advertisements. The CNI merely adds the pod IP to the node’s routing table and optionally configures local policy (e.g., iptables, eBPF). |
Both models ultimately install a route for each pod IP in the node’s main routing table (usually via ip route add <pod‑IP>/32 dev <cni0> or similar). The difference lies in how that route is learned by other nodes and what encapsulation, if any, is applied.
Pod Routing Fundamentals
When a pod is created, the CNI plugin performs the following steps (simplified):
- Address Allocation – Assigns an IP from the pod CIDR (e.g.,
10.244.0.0/16) and optionally a gateway. - Interface Creation – Creates a veth pair; one end lives in the pod namespace (
eth0), the other is attached to a bridge or directly to the host’s network stack. - Route Installation – Adds a host route for the pod IP pointing to the veth peer (or to the bridge). Example on a node:
ip route add 10.244.1.5/32 dev cni0 src 10.244.0.1 - Policy Application – Installs iptables/nftables or eBPF rules for SNAT, DNAT, or network‑policy enforcement.
- Overlay Encapsulation (if applicable) – For overlay CNIs, packets matching the pod IP are handed to an encapsulation device (e.g.,
flannel.1) that adds a VXLAN header and sends them out via the physical NIC.
The forwarding path for a packet originating from Pod A to Pod B is therefore:
Pod A (eth0) → veth pair → host routing table → (encapsulation if overlay) → physical NIC → underlay network → (decapsulation if overlay) → host routing table on destination node → veth pair → Pod B (eth0)Any deviation in the host routing table (e.g., a missing, incorrect, or blackhole route) will break this path and can affect traffic beyond the immediate pod, depending on how the CNI propagates routing information.
Understanding the Blast Radius
Definition and Impact of a Misprogrammed Pod Route
A misprogrammed pod route is any erroneous entry in a node’s routing table that concerns a pod IP address. Common manifestations include:
- Blackhole route (
ip route add <pod‑IP>/32 dev null0) – traffic to the pod is dropped. - Incorrect next‑hop (
ip route add <pod‑IP>/32 dev eth0) – packets are sent to the wrong interface, often causing ARP failures or loops. - Duplicate route (two identical host routes with different metrics) – can cause nondeterministic load‑balancing or flapping.
- Over‑aggregated route (
ip route add 10.244.0.0/16 dev eth0) – hides more specific pod routes, leading to traffic blackholing for all pods in that subnet.
The blast radius is the set of pods, services, or external destinations whose forwarding behavior changes as a direct or indirect consequence of the erroneous route. In the best case the impact is limited to the originating pod; in the worst case it propagates cluster‑wide, causing widespread connectivity loss or routing loops.
Factors Influencing the Blast Radius
| Factor | How It Affects Blast Radius | Example |
|---|---|---|
| CNI Type (Overlay vs Routed) | Overlay CNIs typically confine the effect to the underlay reachability of the encapsulation endpoint; routed CNIs can spread the error via BGP/OSPF to every node. | A blackhole on a flannel node only breaks traffic that must traverse that node’s VTEP; a blackhole advertised via Calico BGP is withdrawn from all peers. |
| Route Propagation Scope | If the CNI injects pod routes into a routing protocol, the error propagates to all protocol peers. Static or locally‑scoped routes stay on the node. | Calico’s bgp mode advertises each pod IP as a /32; a misprogrammed /32 is sent to all BGP peers. |
| Presence of Route Reflectors / Route‑Servers | Centralized reflection can amplify a single bad advertisement to many nodes instantly. | A route‑reflector receiving a bogus host‑route from one node will reflect it to all its clients. |
| ECMP / Load‑Balancing | Misprogrammed routes that participate in ECMP can cause traffic splitting, leading to partial blackholing or micro‑loops. | Two equal‑cost paths to a pod IP; one is a blackhole, causing ~50% loss. |
| Policy Enforcement (iptables/eBPF) | If the misprogrammed route is accompanied by faulty NAT or policy rules, the impact can extend to traffic that never matches the route (e.g., SNAT mis‑translation). | A bad SNAT rule rewrites source IPs of all packets leaving the node, breaking return paths. |
| Underlay Network Characteristics | In overlay CNIs, the underlay must forward encapsulation traffic; a misprogrammed underlay route (e.g., missing default gateway) can affect all overlay traffic. | A flannel node loses its default route → all VXLAN encapsulated packets are dropped. |
| Timer and Convergence Behavior | Slow route withdrawal or dampening can prolong the blast radius; fast convergence limits it. | BGP hold‑time of 180 s keeps a bad route alive longer than necessary. |
Overlay and Routed CNIs
Overview of Overlay CNI
Overlay CNIs create a virtual L2/L3 network that is encapsulated within the physical network. Key components:
- VTEP (Virtual Tunnel End Point) – usually a Linux device like
flannel.1(VXLAN) orvxlan.calico(IP‑in‑IP). - Overlay CIDR – the subnet assigned to pods (e.g.,
10.244.0.0/16). - Underlay – the physical network that carries the encapsulated packets; only needs to route the underlay IPs of the nodes (not pod IPs).
Control Plane – Most overlay CNIs use a simple datastore (etcd, Kubernetes API, or a gossip protocol) to map pod IPs → node IPs → VTEP IPs. When a pod is added, the CNI writes an entry: <pod‑IP> → <node‑IP>. The dataplane looks up the destination node IP, encapsulates the packet, and sends it out.
Failure Containment – Because pod IPs are not advertised into the underlay, a misprogrammed host route on a node only affects traffic that must traverse that node’s VTEP. Traffic that can bypass the node (e.g., via ECMP across multiple nodes) may remain unaffected.
Overview of Routed CNI
Routed CNIs treat pod IPs as regular host routes in the fabric. Essential pieces:
- Host Route –
/32route for each pod IP, installed in the node’s main routing table (ip route add <pod‑IP>/32 dev <cni‑if>). - Routing Protocol – Typically BGP (Calico, Kube‑Router) or OSPF/IS‑IS (less common) that advertises these host routes.
- No Encapsulation – Packets are forwarded natively; the underlay must have reachability to the pod IP.
Control Plane – Each node runs a BGP speaker (or talks to a central route‑reflector). When a pod is added, the CNI injects the host route into the local BGP table; the protocol then propagates it. Withdrawals happen on pod deletion.
Failure Propagation – A mistaken host route (e.g., a blackhole) is treated like any other route and is advertised to peers. Unless mitigated by route‑policy (e.g., reject communities, prefix‑lists), the error can spread to the entire autonomous system (AS) or cluster.
Comparison of Overlay and Routed CNIs
| Aspect | Overlay CNI | Routed CNI |
|---|---|---|
| Address Visibility | Pod IPs are hidden from the underlay; only node IPs are visible. | Pod IPs are visible as host routes in the underlay. |
| Failure Scope | Local to node’s VTEP reachability; overlay traffic may be blackholed if encapsulation fails. | Can propagate via routing protocol to all nodes; a blackhole affects all traffic to that pod. |
| Complexity | Simpler underlay (just needs L2/L3 connectivity). Requires encapsulation/dataplane overhead. | Requires a routing protocol stack; more complex policy to prevent leaks. |
| Scalability | Limited by overlay table size (VTEP mappings) and encapsulation overhead; generally scales to thousands of nodes. | Scales with routing protocol capacity; BGP can handle hundreds of thousands of routes if properly tuned. |
| Typical Use Cases | Simple clusters, cloud providers that discourage BGP, environments where pod IPs must remain private. | Bare‑metal, on‑prem, or hybrid clouds where network teams already run BGP and desire visibility/traffic engineering. |
| Example Plugins | Flannel (VXLAN), Weave Net, Calico (IP‑in‑IP), Canal. | Calico (BGP mode), Cilium (native routing), Kube‑Router, Romana. |
Mapping the Blast Radius
Identifying Containment Boundaries
Containment boundaries are points in the forwarding plane where a misprogrammed route cannot influence traffic beyond. They depend on the CNI model:
| CNI Model | Containment Boundary | Reason |
|---|---|---|
| Overlay | Encapsulation endpoint (VTEP) on the source node. If the VTEP forwards the encapsulated packet correctly, the error stays local. | The underlay only sees node IPs; a pod‑IP misroute does not affect underlay forwarding. |
| Overlay | Underlay reachability of the VTEP destination. If the underlay cannot reach the remote node’s VTEP IP, all overlay traffic to/from that node is affected – this is a broader boundary. | Misprogrammed underlay route (e.g., missing default gateway) impacts every encapsulated flow. |
| Routed | Local routing table of the node where the misroute exists. If the route is not advertised (e.g., suppressed by a route‑policy), the blast radius stays on that node. | No protocol propagation → only locally sourced/destined traffic sees the error. |
| Routed | Routing protocol adjacency (BGP peer, OSPF neighbor). If the erroneous route is filtered out before being sent to peers, the blast radius is limited to the local AS or area. | Route‑maps, prefix‑lists, or reject communities can stop propagation. |
| Routed | ECMP group – if the misroute is one of several equal‑cost paths, only a fraction of traffic is impacted. | Load‑balancing dilutes the effect. |
Analyzing Failure Propagation
To map the blast radius we follow the control‑plane trace:
- Origin – A CNI plugin writes an incorrect host route (
ip route add <pod‑IP>/32 dev null0). - Local Effect – Packets destined for
<pod‑IP>are dropped locally; traffic from the pod may still egress if source‑based routing is not used. - Propagation Decision –
- Overlay: No propagation; the route stays in the node’s FIB.
- Routed: The route is inserted into the local BGP/OSPF process.
- Protocol Transmission – The routing protocol advertises the route to peers (BGP UPDATE, OSPF LSA).
- Peer Acceptance – Peers install the route in their RIB/FIB unless blocked by inbound policy.
- Forwarding Consequence – Any node that now has a blackhole for
<pod‑IP>will drop packets destined for that pod, regardless of where the packet originated. - Potential Amplification – If the pod IP is part of a larger aggregate (e.g., a /24 subnet advertised as a summary), the blackhole may attract traffic for all pods in that aggregate, multiplying the impact.
The amplification factor can be approximated as:
Amplification = (Number of nodes that install the bogus route) × (Fraction of traffic that matches the bogus route)Understanding these boundaries and propagation mechanisms allows operators to design safeguards—such as route‑filtering, BGP communities, or underlay redundancy—to limit the blast radius of a misprogrammed pod route.