NodePort resets under overlay and asymmetric return paths

Introduction to NodePort Connection Resets

Overview of NodePort Service Type

A NodePort service exposes a TCP/UDP port on every node’s IP at a static port (the nodePort). Traffic arriving at <node-ip>:<nodePort> is routed by kube‑proxy to a backend pod. The service abstraction adds a layer of iptables/IPVS rules that perform destination NAT (DNAT) to the pod’s cluster IP and source NAT (SNAT) when externalTrafficPolicy is set to Cluster (the default).
When a client connects, the SYN packet is DNAT’d to a pod; the reply traverses the same path in reverse. If any step corrupts the 5‑tuple or drops the packet, the client sees a TCP reset (RST) instead of a successful handshake.

Common Causes of Connection Resets

Observed symptoms are identical for several distinct failure modes:

TCP RST in client logs or tcpdump output.
No SYN‑ACK received after SYN, or an immediate RST after SYN‑ACK.
Intermittent behavior that correlates with node changes, pod rescheduling, or iptables reload.

The four most frequent culprits are:

Incorrect SNAT (source address altered when it should be preserved).
Reverse‑path filtering (rp_filter) dropping reply packets because the kernel deems the source address unreachable on the ingress interface.
Wrong interface assumptions (e.g., service bound to eth0 while traffic arrives over eni or a VLAN sub‑interface).
Genuine service‑proxy defects (bugs in kube‑proxy/IPVS or the CNI’s host‑local SNAT implementation).

Each hypothesis produces the same outward symptom, so we must isolate them with targeted diagnostics.

Troubleshooting NodePort Connection Resets

Identifying SNAT Mistakes

Configuring SNAT Correctly

SNAT is applied by kube‑proxy when externalTrafficPolicy: Cluster. The node’s outgoing packet (from pod to client) gets the node’s primary address as source. If the client expects the original source IP (e.g., for logging or ACLs) and receives a different address, some middleboxes or the client TCP stack may reject the packet with RST.
When externalTrafficPolicy: Local is set, kube‑proxy skips the node‑level SNAT step; the packet leaves the pod with the pod’s IP as source. This eliminates the SNAT‑induced mismatch but introduces the constraint that traffic must arrive at the node hosting the endpoint.

Example SNAT Configuration using CLI

# Example CLI command to configure SNAT
kubectl patch svc <service-name> -p '{"spec":{"externalTrafficPolicy":"Local"}}'

What this does: forces kube‑proxy to install iptables/IPVS rules that omit the MASQUERADE rule for traffic destined to the NodePort.
How to test:

Capture traffic on the client with tcpdump -i any -nn port <nodePort>.
Verify that the SYN‑ACK’s source IP matches the pod IP (not the node IP).
If the client still resets, SNAT is not the root cause.

Reverse-Path Filtering Issues

Understanding Reverse-Path Filtering

Linux rp_filter (enabled by default in many distributions) performs a source‑address validation on incoming packets: the kernel checks whether the reverse path (the route to the packet’s source address) would use the same interface on which the packet arrived. If the check fails, the packet is silently dropped.
In overlay networks (e.g., VXLAN, Geneve) the pod’s IP may be reachable via a tunnel interface (vxlan.calico, veth0, etc.) while the NodePort traffic arrives on the physical NIC (eth0). The reply from the pod traverses the tunnel, but the client’s reply (the ACK of the SYN‑ACK) arrives on eth0. If rp_filter is set to 1 (strict mode) on eth0, the kernel sees the source address (pod IP) as unreachable via eth0 and drops the ACK, causing the client to retransmit SYN and eventually reset.

Disabling Reverse-Path Filtering

# Example CLI command to disable reverse-path filtering
sysctl -w net.ipv4.conf.all.rp_filter=0

What this does: turns off strict reverse‑path verification for all interfaces.
How to test:

Check current value: sysctl net.ipv4.conf.eth0.rp_filter.
If it is 1, set to 0 and repeat the connection attempt.
Use tcpdump -i eth0 -nn to see whether the ACK now arrives.
If the connection succeeds, the issue was rp_filter.
Caveat: disabling rp_filter can increase susceptibility to spoofing attacks; re‑enable with a more granular setting (net.ipv4.conf.eth0.rp_filter=2 for loose mode) if your topology permits.

Wrong Interface Assumptions

Identifying Correct Network Interfaces

NodePort services bind to the node’s primary IP address by default. In multi‑homed nodes (multiple NICs, VLANs, or cloud provider elastic network interfaces) the node may have several IPs. If a client reaches the service via a secondary IP that is not advertised by the node’s kube‑proxy (e.g., because the service’s externalIPs field is empty and the cloud load balancer forwards to a secondary IP), kube‑proxy will still DNAT the packet to a pod, but the reply will be sourced from the primary IP. The client sees a mismatch between the destination IP it sent to and the source IP of the reply, which can trigger a reset if the client performs strict address validation (some HTTP clients, proxies, or stateful firewalls do this).

Configuring Correct Network Interfaces

# Example CLI command to configure network interfaces
kubectl patch svc <service-name> -p '{"spec":{"externalTrafficPolicy":"Local","loadBalancerIP":"<load-balancer-ip>"}}'

What this does: forces the service to use a specific IP (loadBalancerIP) as the outward‑facing address. When combined with externalTrafficPolicy: Local, it tells kube‑proxy to install DNAT rules only on the interface that owns that IP.
How to test:

Determine which interface carries the client‑targeted IP: ip addr show | grep <load-balancer-ip>.
Verify that kube‑proxy has installed rules for that interface: iptables -t nat -L KUBE-NODEPORTS -v -n | grep <load-balancer-ip>.
If rules are missing on the interface in use, the patch will cause kube‑proxy to create them.
Retry the connection; a successful handshake indicates the interface mismatch was the problem.

Genuine Service-Proxy Defects

Identifying Service-Proxy Issues

When the above three categories are ruled out, the residual cause is a bug in the proxy implementation (kube‑proxy in iptables mode, IPVS mode, or the CNI’s host‑local SNAT). Defects can manifest as:

Incorrect handling of the TCP state machine (e.g., sending RST instead of ACK during SYN‑RECEIVED).
Race conditions where a service endpoint is removed while a packet is in flight, causing the proxy to send a RST with the “connection refused” flag.
Mis‑calculation of the checksum or segmentation offload leading to a corrupted packet that the receiver treats as invalid.

Debugging Service-Proxy Configuration

# Example CLI command to debug service-proxy configuration
kubectl logs <service-proxy-pod-name> -f

What this does: streams the logs of the kube‑proxy pod (or the CNI’s node‑agent if it implements proxying). Look for lines such as:

E0925 10:15:42.123456       1 proxier.go:456] failed to sync service <svc>: error updating iptables rule: ...

I0925 10:15:42.987654       1 ipvs.go:123] Adding destination <pod-ip>:<targetPort> for service <svc>

How to test:

Reproduce the failure while capturing logs.
Correlate timestamps of RST packets (from tcpdump) with proxy log entries.
If you see repeated “failed to sync” or “endpoint not found” messages at the moment of the reset, the proxy is likely at fault.
As a workaround, restart the proxy (kubectl rollout restart daemonset/kube-proxy -n kube-system) or switch modes (--proxy-mode=ipvs vs iptables) to see if the issue persists.

Code Examples for NodePort Configuration

Example NodePort Service YAML Configuration

apiVersion: v1
kind: Service
metadata:
  name: <service-name>
spec:
  selector:
    app: <app-name>
  ports:
  - name: http
    port: 80
    targetPort: 8080
    nodePort: 30080
    protocol: TCP
  type: NodePort

Example CLI Command to Create NodePort Service

# Example CLI command to create NodePort service
kubectl create -f <service-yaml-file>

Scaling Limitations of NodePort Services

NodePort Service Scaling Considerations

Each node can listen on a given nodePort only once. If you run many replicas of a service, kube‑proxy must install DNAT rules that load‑balance across all endpoints. In iptables mode this results in a linear chain of rules; latency grows with the number of endpoints. IPVS mitigates this with hash tables, but the kernel still maintains a per‑service state array that can become a memory pressure point on large clusters (>5000 endpoints per service).
Additionally, because the service is bound to every node’s IP, the client’s load‑balancing is naive: traffic is distributed per‑connection based on the node that answered the ARP request, not based on actual service load. This can lead to hotspots when a subset of nodes runs the majority of pods.

Load Balancer Integration for Scalability

To obtain true load‑balancing and health‑checking, place a cloud L4 LB or an external hardware LB in front of the NodePort addresses. The LB forwards to <node-ip>:<nodePort> on a pool of nodes; the LB can perform session persistence, health checks, and dynamic scaling without touching kube‑proxy.

# Example CLI command to integrate load balancer
kubectl patch svc <service-name> -p '{"spec":{"loadBalancerIP":"<load-balancer-ip>"}}'

Note: The loadBalancerIP annotation is only honored by certain cloud providers (e.g., GKE, AKS). On bare‑metal you must rely on an external LB that points to the node IPs.

Service Proxy Performance Optimization

Optimizing Service Proxy Configuration

Switch to IPVS (kube-proxy --proxy-mode=ipvs) for O(1) lookup.
Enable ipvs.scheduler (rr, lc, wlc, lblc, lblcr, dh, sh, sed, nq) to match your traffic pattern.
Adjust net.ipv4.vs.conntrack and net.ipv4.vs.expire_nodst_conn to keep connection tracking tables from overflowing.
Disable unnecessary conntrack helpers (net.netfilter.nf_conntrack_helper=0) if appropriate for your workload.