From Node CIDR Allocation to Pod Route Advertisement

Introduction to Routed CNI

A routed Container Network Interface (CNI) treats each pod’s IP address as a routable prefix in the underlying IP fabric. Instead of overlay encapsulation (VXLAN, Geneve), the CNI programs the host’s routing table with a /32 (IPv4) or /128 (IPv6) host route pointing to the pod’s veth pair, then advertises the pod’s allocated CIDR (typically a /24 per‑node block) via BGP to upstream routers. Forwarding follows native L3 routing:

1. Pod → veth pair → host network namespace – packet leaves the pod via its veth interface.
2. Host routing table – a host‑specific /32 route directs the packet to the veth peer.
3. Node’s default route / upstream BGP‑learned route – the packet is forwarded toward the destination pod’s node using the advertised pod CIDR.
4. Remote node – the same host‑route mechanism delivers the packet to the target pod.

Key Control‑Plane Objects

• Kubernetes Pod CIDR allocation (controller manager or external IPAM).
• CNI plugin – creates the veth pair, assigns the pod IP, installs host routes.
• BGP speaker – redistributes the node‑local pod CIDR into the fabric (often embedded in the CNI or run as a sidecar).
• Upstream routers – install the advertised CIDR into their FIB and propagate it further.

Kubernetes Allocation of Pod CIDR

When a node registers, the node-controller (part of kube‑controller-manager) ensures the Node object’s spec.podCIDR field is populated.

Allocation Process

1. Node registration – kubelet sends a Node object (via TLS‑bootstrap or existing credentials) to the apiserver; the object initially lacks spec.podCIDR.
2. Node CIDR controller – watches for Node objects missing spec.podCIDR. It calculates a free subnet from the cluster‑wide --cluster-cidr (e.g., 10.244.0.0/16).
3. Allocation algorithm – the controller maintains an in‑memory bitmap of allocated /24 blocks (or the size defined by --node-cidr-mask-size). It picks the first free block and attempts an atomic UPDATE on the Node object via the apiserver (conditional update to avoid races).
4. Persistence – the apiserver writes the updated Node object to etcd; spec.podCIDR now holds the allocated prefix (e.g., 10.244.1.0/24).
5. Node startup – kubelet reads the Node object (via downward API or a watch) and passes the CIDR to the CNI plugin through the KUBERNETES_POD_CIDR environment variable or the CNI ipam plugin’s subnet field.

If an external IPAM (e.g., Infoblox, NetBox, or a custom CRD) is used, the allocation flow diverges: the IPAM controller creates a PodNetwork or IPPool CRD, updates the Node’s annotations or a dedicated NodeIPAM object, and the kubelet’s CNI shim queries the IPAM via a gRPC/REST endpoint.

API and etcd Storage

• API endpoint: GET /api/v1/nodes/{node-name} returns the Node object; spec.podCIDR is part of NodeSpec.
• etcd key: /registry/nodes/{node-name} stores the serialized Node object (protobuf); podCIDR appears as a string under the spec section.
• Watch mechanism: kubelet establishes a watch on /api/v1/nodes?fieldSelector=metadata.name={node-name} to receive updates instantly when the CIDR is populated.
• Garbage collection: When a node is deleted, the node-controller releases the CIDR back into the free pool for future nodes.

CNI Plugin for Pod Networking

The CNI spec defines a simple contract: an executable invoked with CNICOMMAND=ADD|DEL|VERSION and a JSON payload on stdin. The plugin must:

• Allocate an IP address for the pod (via its own IPAM or by delegating to a plugin like host-local).
• Create a veth pair, move one end into the pod’s network namespace, and assign the IP.
• Configure routes in the host namespace (typically a host‑specific /32 route to the veth peer).
• Optionally invoke iptables, ipvs, or ebpf for service handling.
• Return a Result JSON containing the assigned IP, interfaces, and routes.

Types Relevant to Routed CNI

CNI Configuration and Pod Networking Setup

A typical routed CNI configuration (Calico‑style) resides in /etc/cni/net.d/10-calico.conflist:

{
  "cniVersion": "0.4.0",
  "name": "k8s-pod-network",
  "plugins": [
    {
      "type": "calico",
      "log_level": "info",
      "ipam": {
        "type": "host-local",
        "subnet": "usePodCidr"
      },
      "policy": {
        "type": "k8s"
      },
      "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/__kubeconfig__"
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    }
  ]
}

Key fields

• "type": "calico" invokes the Calico binary (calico).
• "ipam": { "type": "host-local", "subnet": "usePodCidr" } tells the plugin to read the pod CIDR from the environment variable KUBERNETES_POD_CIDR (set by kubelet).
• The portmap plugin enables hostPort mapping.

When kubelet executes the CNI ADD command:

1. Environment: KUBERNETES_POD_CIDR=10.244.1.0/24 (from Node object).
2. IPAM call: host-local allocates an IP from that subnet, e.g., 10.244.1.42.
3. Veth creation: vethabc123 (host) ↔ vethdef456 (pod ns).
4. IP assignment: ip addr add 10.244.1.42/32 dev vethdef456 inside pod ns; ip link set vethdef456 up.
5. Host route: ip route add 10.244.1.42/32 dev vethabc123.
6. Result JSON returned to kubelet includes the allocated IP and interfaces.

Example CNI Configuration Files and Code Snippets

Minimal host‑gw CNI config (Flannel):

{
  "cniVersion": "0.4.0",
  "name": "flannel",
  "type": "flannel",
  "delegate": {
    "isDefaultGateway": true
  }
}

Calico CNI plugin pseudo‑code (simplified):

func cmdAdd(args *skel.CmdArgs) error {
    // 1. Parse CNI args
    var netConf types.NetConf
    if err := json.Unmarshal(args.StdinData, &netConf); err != nil {
        return err
    }

    // 2. Retrieve pod CIDR from env
    podCIDR, ok := os.LookupEnv("KUBERNETES_POD_CIDR")
    if !ok {
        return fmt.Errorf("KUBERNETES_POD_CIDR not set")
    }

    // 3. IPAM allocation (host-local)
    ipamResult, err := ipam.ExecAdd(netConf.IPAM.Type, args.StdinData)
    if err != nil {
        return err
    }
    podIP := ipamResult.IPs[0].Address.IP // e.g., 10.244.1.42/32

    // 4. Create veth pair
    hostVeth, containerVeth, err := ip.SetupVeth(args.IfName, args.Netns, podIP)
    if err != nil {
        return err
    }

    // 5. Move container end into ns and configure
    if err := netns.WithNetNSPath(args.Netns, func(_ ns.NetNS) error {
        if err := ip.AddrAdd(containerVeth, podIP, 32); err != nil { return err }
        if err := ip.LinkSetUp(containerVeth); err != nil { return err }
        if err := ip.LinkSetLoUp(); err != nil { return err }
        return nil
    }); err != nil { return err }

    // 6. Host‑side route (host‑specific /32)
    if err := ip.AddRoute(podIP.String()+"/32", hostVeth, ""); err != nil { return err }

    // 7. Return result
    result := &types.Result{
        CNIVersion: types.Version,
        Interfaces: []*types.Interface{
            {Name: args.IfName, Mac: hostVeth.HardwareAddr.String(), Sandsbox: ""},
            {Name: args.IfName, Mac: containerVeth.HardwareAddr.String(), Sandsbox: args.Netns},
        },
        IPs: []*types.IPConfig{
            {Version: "4", Address: podIP, Gateway: net.IP{}},
        },
        Routes: []*types.Route{
            {Dst: net.IPNet{IP: podIP, Mask: net.CIDRMask(32, 32)}, GW: net.IP{}},
        },
    }
    return types.PrintResult(result, args.CNIVersion)
}

BGP Advertisement of Pod CIDR

BGP (RFC 4271) exchanges NLRI (Network Layer Reachability Information) via UPDATE messages. Relevant message types for routed CNI:

• OPEN – establishes session, negotiates version, ASN, hold time, capabilities (e.g., multiprotocol, route‑refresh).
• UPDATE – advertises or withdraws NLRI; carries path attributes (AS_PATH, NEXT_HOP, LOCAL_PREF, MED, COMMUNITY, etc.).
• KEEPALIVE – maintains session.
• NOTIFICATION – signals errors.

In a routed CNI, the NEXT_HOP attribute is typically set to the node’s own IP (the BGP speaker’s router ID or a loopback). The NLRI is the pod CIDR (e.g., 10.244.1.0/24). No encapsulation is involved; the upstream router uses the NEXT_HOP to resolve the path via its own IGP or static routes.

BGP Peering and Route Advertisement

Peering options:

• Node‑to‑leaf – each node runs a BGP speaker that peers directly with the top‑of‑rack (ToR) switch.
• Node‑to‑route‑reflector – nodes peer with a central RR (often a pair of route‑reflectors for redundancy).
• Hybrid – nodes peer with both leaf and RR for load‑sharing.

Advertisement process:

1. BGP speaker startup – reads the node’s pod CIDR from an environment variable, file, or CNI plugin via a Unix socket.
2. Session establishment – sends OPEN to peer; upon successful ESTABLISHED state, begins exchanging routes.
3. Route origination – the speaker generates an UPDATE with NLRI = pod CIDR, NEXT_HOP = its own BGP identifier (often the node’s primary IP), and optionally sets LOCAL_PREF = 100, MED = 0.
4. Withdrawal – when a node is drained or the CNI plugin detects that the pod CIDR is no longer locally valid (e.g., node deletion), it sends an UPDATE with the same NLRI marked as withdrawn.
5. Route‑reflector handling – the RR reflects the UPDATE to its clients, preserving the ORIGINATOR_ID and CLUSTER_LIST attributes to prevent loops.

Example BGP Configuration and CLI Commands

FRR (Free Range Routing) configuration on a node (/etc/frr/frr.conf):

!
! Define the router ID (use the node's primary IP)
router-id 10.0.5.23
!
! BGP process
router bgp 65012
  bgp router-id 10.0.5.23
  neighbor 10.0.0.1 remote-as 65000   ! ToR leaf
  neighbor 10.0.0.1 activate
  neighbor 10.0.0.1 capability extended