Hairpin service failures when a Pod calls itself

Introduction to Pod-to-Service VIP Communication

Overview of Kubernetes Networking

Kubernetes provides a flat pod CIDR; each pod receives an IP from this range and can reach any other pod’s IP without NAT. Services expose a stable virtual IP (VIP) – the cluster IP – that load‑balances traffic to backend pods. The VIP is implemented by kube‑proxy (iptables, IPVS, or eBPF) and resides in the node’s network namespace.

Service VIP and Pod Communication

When a pod sends a packet to its own Service VIP, the packet traverses the node’s networking stack twice:

1. Outbound – pod → veth → bridge → … → kube‑proxy → DNAT to a backend pod IP.
2. Inbound – the reply follows the reverse path.

If kube‑proxy selects the originating pod as the backend, the traffic must hairpin: go out via the bridge and back in to the same veth pair. Without proper hairpin handling the bridge drops the frame, causing a connection timeout or reset.

Hairpin Handling in Kubernetes

Definition and Purpose

Hairpin handling (also called hairpin mode or promiscuous bridge) allows a Linux bridge to forward a frame received on one port back out through the same port when the destination MAC matches the port’s own MAC. In the pod‑to‑service‑VIP case, the bridge must hairpin the Ethernet frame that carries the IP packet destined for the pod’s own MAC after DNAT.

If hairpin is disabled, the bridge treats the frame as ingress and egress on the same port and discards and discards it to prevent loops, leaving the pod with no reply.

Configuring Hairpin Handling

Hairpin is a property of the CNI bridge used by the node. Most CNIs expose a toggle:

Enabling hairpin tells the CNI to set the bridge’s hairpin flag via brctl setfd / ip link set dev <bridge> hairpin on.

Example Configuration

Plain bridge CNI (hairpin enabled)

{
  "cniVersion": "0.4.0",
  "name": "kubenet",
  "type": "bridge",
  "bridge": "cni0",
  "isGateway": true,
  "ipMasq": false,
  "hairpinMode": true,
  "ipam": {
    "type": "host-local",
    "subnet": "10.244.0.0/16",
    "routes": [{ "dst": "0.0.0.0/0" }]
  }
}

Flannel VXLAN backend with hairpin

{
  "name": "cbr0",
  "cniVersion": "0.4.0",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": { "portMappings": true }
    }
  ]
}

After the CNI is restarted (or the node is rebooted), verify the bridge flag:

# Assuming the bridge is named cni0
brctl show hairpin cni0
# Expected output: hairpin flag: on
# or with iproute2:
ip link show cni0 | grep hairpin
#  <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
#      link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
#      hairpin on

Masquerade Rules and Their Impact on Pod-to-Service VIP Traffic

Understanding Masquerade Rules

Kube‑proxy in iptables mode installs NAT rules in the nat table. For traffic destined to a Service VIP, a DNAT rule rewrites the destination IP to a selected endpoint pod IP. Return traffic is un‑DNAT’ed by connection‑tracking (conntrack).

When traffic leaves the node (e.g., to reach a pod on a different node), kube‑proxy adds a MASQUERADE rule in the POSTROUTING chain to replace the source IP with the node’s primary address. This is required because the pod’s IP is not routable outside the cluster.

For local traffic (pod and service VIP on the same node) the MASQUERADE rule is not needed; the source IP can stay as the pod IP. However, if the MASQUERADE rule is overly broad (e.g., matches all traffic to the cluster CIDR), it will also SNAT the hairpinned packet, causing the reply to be sourced from the node address instead of the pod IP. The originating pod may then drop the reply because it does not match the expected source address (or the conntrack entry is mismatched).

Configuring Masquerade Rules

The masquerade scope is controlled by kube‑proxy’s --masquerade-all flag or, more granularly, by the iptables rule comment containing KUBE-MARK-MASQ. In IPVS mode, masquerade is handled by the IPVS sync daemon; the same principles apply.

To restrict masquerade to off‑node traffic only, adjust the rule to match packets where the source is not a local pod CIDR and the destination is a Service VIP. Many CNIs (e.g., Calico) install a KUBE-MARK-MASQ chain that already does this check.

Inspecting the MASQUERADE rule set

# List nat rules related to kube‑proxy
sudo iptables -t nat -L -v -n | grep -E "KUBE-MARK-MASQ|MASQUERADE"

Typical output (iptables mode):

Chain KUBE-MARK-MASQ (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x0/0xffff

The rule marks packets for SNAT only if they originate from non‑local addresses (the mark is later used by the KUBE-POSTROUTING chain). If you see a rule that matches 0.0.0.0/0 without a source restriction, masquerade will be applied indiscriminately.

CLI Examples for Masquerade Rule Configuration

To temporarily disable masquerade for a test (note: this will break external connectivity; restore after testing):

# Identify the rule number in KUBE-POSTROUTING
sudo iptables -t nat -L KUBE-POSTROUTING --line-numbers
# Suppose line 2 is the blanket MASQUERADE
sudo iptables -t nat -D KUBE-POSTROUTING 2

To make a persistent change, edit the kube‑proxy ConfigMap (for kubeadm clusters):

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-proxy
  namespace: kube-system
data:
  config.conf: |
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    kind: KubeProxyConfiguration
    mode: "iptables"
    masqueradeAll: false   # <-- disables global masquerade
    # masqueradeBit: 14   # optional, default

After updating the ConfigMap, roll out the daemon set:

kubectl rollout restart daemonset/kube-proxy -n kube-system

Verify that masquerade is now conditional:

sudo iptables -t nat -L KUBE-POSTROUTING -v -n
# Expect to see a rule that matches ! -s 10.244.0.0/16 (pod CIDR) before MASQUERADE

Local Bridge Assumptions and Their Role in Pod-to-Service VIP Communication

Local Bridge Concept

Each node typically creates a Linux bridge (e.g., cni0, br0, or docker0) that connects the veth pairs of pods to the node’s root namespace. The bridge operates at layer 2: it forwards Ethernet frames based on MAC address learning.

When a pod sends a packet to a Service VIP on the same node:

1. Pod’s veth → bridge (lookup destination MAC).
2. If the MAC is unknown, the bridge floods the frame to all ports except the ingress port.
3. The frame reaches the veth of the kube‑proxy “dummy” interface or the node’s primary interface (depending on CNI).
4. IP tables/IPVS DNAT rewrites the destination to a backend pod IP.
5. The packet is routed (or bridged) back to the veth of the selected backend pod.
6. The reply traverses the reverse path.

If the selected backend pod is the same pod that originated the request, step 5 attempts to send the frame out the same veth port it arrived on. The bridge must therefore hairpin the frame.

Assumptions and Limitations

Troubleshooting Local Bridge Issues

1. Verify the bridge hairpin flag

   # Replace cni0 with your bridge name
   ip link show cni0 | grep hairpin
   # Expected: hairpin on

2. Check bridge forwarding database (FDB) for the pod’s MAC

   # Find the pod’s MAC (inside the pod)
   POD_MAC=$(kubectl exec -it <pod> -- cat /sys/class/net/eth0/address)
   # Look it up in the bridge FDB
   bridge fdb show dev cni0 | grep $POD_MAC
   # Should show the port (vethXXX) as destination

3. Capture traffic on the veth pair

   # On the node, identify the veth paired with the pod
   # Example using `docker`/`crictl` to get sandbox ID then `ethtool`
   sudo tcpdump -i vethXXX -e -nn icmp

   Look for:

   • Outgoing ICMP echo request with correct source MAC (pod) and destination MAC (bridge).
   • If the request leaves the veth but no reply returns, the bridge is likely dropping the hairpinned frame.

4. Validate conntrack entries

   sudo conntrack -L -p icmp --src <pod_ip> --dst <service_vip>

   Expect to see a single entry in ESTABLISHED state. If you see SNAT altering the source address, masquerade is mis‑applied.

5. Test with hairpin disabled/enabled

   # Disable hairpin temporarily
   sudo ip link set dev cni0 hairpin off
   # Retry the test; expect failure
   sudo ip link set dev cni0 hairpin on
   # Retry; expect success

Troubleshooting Pod-to-Service VIP Communication Breakdowns

Identifying Common Issues

Using CLI Tools for Troubleshooting

• kubectl exec – run diagnostics inside the pod (curl, wget, nc -vz).
• nsenter – enter the host network namespace to inspect bridges, iptables, conntrack.
• tcpdump / bpftrace – capture at veth, bridge, or host interface.
• iptables -t nat -L -v -n – verify DNAT and MASQUERADE rules.
• ip route get – confirm the packet is routed locally (should show dev lo or dev <bridge>).
• conntrack -L – track state of the flow.

Example Troubleshooting Scenarios

Scenario 1 – Hairpin disabled after node reboot

Symptom: Pod A cannot reach its own Service VIP; curl times out after ~10 s.
Investigation:

# On the node
brctl show hairpin cni0
# Output: hairpin flag: off

Fix: Ensure the CNI conf has hairpinMode: true and reload the CNI (or restart the node).

Scenario 2 – Masquerade applied to local traffic

Symptom: curl returns quickly but the response source IP is the node’s IP, causing the pod to drop the packet (observed as Connection reset by peer).
Investigation:

sudo iptables -t nat -L KUBE-POSTROUTING -v -n | grep MASQUERADE

If the rule lacks a source restriction (e.g., matches 0.0.0.0/0), masquerade is being applied to hairpinned traffic.

Fix: Adjust the kube‑proxy ConfigMap to set masqueradeAll: false or refine the KUBE-MARK-MASQ chain to exclude the local pod CIDR, then restart kube‑proxy.

End of document.