Why one backend stopped receiving service traffic

Introduction to EndpointSlices and ClusterIP

Overview of EndpointSlices

EndpointSlices are a scalable, partitioned replacement for the legacy Endpoints object in Kubernetes. Each slice holds a limited number of endpoints (default 100) and is identified by the label kubernetes.io/service-name. The controller manager creates and updates slices in response to changes in Pod readiness, node affinity, or topology spread constraints. kube-proxy watches these slices on each node to program IPVS or iptables rules that direct ClusterIP traffic to healthy backends.

Role of ClusterIP in Kubernetes

A Service of type ClusterIP allocates a virtual IP from the cluster’s service CIDR. Clients inside the cluster resolve the service name via CoreDNS to that IP. kube-proxy translates the virtual IP to a set of destination addresses derived from the current EndpointSlices for that service. The translation occurs per‑node, using either iptables (NAT mode) or IPVS (proxy mode). If the node’s view of the EndpointSlices lags behind the control plane, traffic may be blackholed for a subset of backends even though the slices appear updated globally.

Understanding the Failure Scenario

Initial Symptoms and Observations

A deployment of three replicas (app=frontend) exposes a ClusterIP service svc/frontend.
Clients intermittently receive connection timeouts to svc/frontend:8080.
kubectl get endpointslices -l kubernetes.io/service-name=frontend shows three slices, each with two ready addresses (total six endpoints) matching the three Pods.
kubectl get pods -l app=frontend reports all Pods Ready: 1/1.
kube-proxy logs on the affected node show no error messages; IPVS sync appears successful.
curl -v svc/frontend:8080 from a pod on the same node sometimes hangs, while the same command from a different node succeeds.

Investigating EndpointSlice Updates

Verify slice version numbers:

kubectl get endpointslices -l kubernetes.io/service-name=frontend -o yaml | grep -E 'metadata:|resourceVersion:'

Output shows all slices at resourceVersion: "158423" – identical across nodes.

Check the slice’s addressType and ports:

kubectl get endpointslices -l kubernetes.io/service-name=frontend -o jsonpath='{.items[*].addressType}{"\n"}'

Returns IPv4 for all slices.

Confirm that each slice contains the expected Pod IPs:

for slice in $(kubectl get endpointslices -l kubernetes.io/service-name=frontend -o name); do
  echo "$slice:"; kubectl get $slice -o jsonpath='{.endpoints[*].addresses}{"\n"}'
done

Lists six distinct IPs, matching the three Pods (each Pod appears twice due to slice partitioning).

At this point the control‑plane view appears healthy.

Identifying the Silent Backend Failure

Run a continuous test from a pod on the suspect node:

while true; do
  timeout 2 bash -c "</dev/tcp/$(kubectl get svc frontend -o jsonpath='{.spec.clusterIP}')/8080" && echo "$(date) OK" || echo "$(date) FAIL"
  sleep 0.5
done

Output shows a pattern of FAIL every ~30 seconds, correlating with the node’s kube-proxy resync interval.

Dump the node’s IPVS table:
```
ipvsadm -Ln
```
Shows only four of the six expected backend IPs; the missing two belong to a single Pod (frontend-5c9d6f7b9-9gk2t).
Compare with another node’s IPVS table – all six IPs present.

Thus, one backend is silently excluded from the node’s load‑balancing dataplane despite the EndpointSlice being up‑to‑date in etcd.

Troubleshooting the Issue

Checking Node‑Local Rendering Lag

kube-proxy watches EndpointSlices via the Kubernetes API and applies changes locally. The watch can be delayed by:

High API server load causing throttling.
Large number of slices causing slow diff computation.
Node‑local caching (e.g., iptables-restore or IPVS sync batch) that only applies changes on a periodic full resync (--sync-period, default 30 s).

To verify lag:

Enable verbose kube-proxy logs on the node:

kubectl -n kube-system logs -l k8s-app=kube-proxy --tail=100 | grep -E 'endpointSlice|sync'

Look for lines like:

I0925 14:03:12.345678       1 reflector.go:205] Watching *v1.EndpointSlice
I0925 14:03:12.345679       1 proxier.go:456] EndpointSlice svc/frontend: received update, version 158423
I0925 14:03:12.345685680       1 proxier.go:470] Scheduling sync in 20s (syncPeriod=30s)

The gap between receipt and scheduled sync indicates the rendering lag.

Check the node’s kube-proxy config:
```
kubectl -n kube-system get configmap kube-proxy -o yaml | grep syncPeriod
```
If syncPeriod is set to a high value (e.g., "5m"), updates are delayed.

Verifying Service State and EndpointSlices

Confirm the Service object has not been mutated:
```
kubectl get svc frontend -o yaml | grep -E 'clusterIP:|sessionAffinity:'
```
Returns the expected clusterIP: 10.96.123.45 and sessionAffinity: None.
Ensure no ExternalTrafficPolicy or healthCheckNodePort is interfering:
```
kubectl get svc frontend -o jsonpath='{.spec.externalTrafficPolicy}{"\n"}'
```
Returns <empty> (default Cluster).

Using CLI Tools for Debugging

Expose kube-proxy metrics (if enabled):
```
kubectl get --raw /api/v1/nodes/worker-03/proxy/metrics | grep kubeproxy
```
Look for kubeproxy_sync_latency_seconds spikes.

Inspect the netns of the kube-proxy container and run ipvsadm directly:

NODE=worker-03
POD=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy -o jsonpath="{.items[?(@.spec.nodeName=='$NODE')].metadata.name}")
kubectl -n kube-system exec $POD -- nsenter --target 1 --mount --uts --ipc --net -- ipvsadm -Ln

Code Examples for Troubleshooting

Using `kubectl` to Inspect EndpointSlices

# List all slices for a service with resource versions
kubectl get endpointslices -l kubernetes.io/service-name=frontend -o wide

# Show a single slice in JSON, including endpoints
kubectl get endpointslice frontend-abcdefg -o json |
  jq '.endpoints[].addresses, .endpoints[].conditions.ready'

Checking Service State with `kubectl get`

# Verify ClusterIP and port
kubectl get svc frontend -o jsonpath='{.spec.clusterIP}{"\t"}{.spec.ports[0].port}{"\n"}'

# Confirm no stray selectors
kubectl get svc frontend -o jsonpath='{.spec.selector}'

Example CLI Commands for Debugging

# Watch EndpointSlice updates in real time
kubectl get endpointslices -l kubernetes.io/service-name=frontend -w

# Dump kube-proxy IPVS rules on a node
kubectl -n kube-system exec $(kubectl -n kube-system get pod -l k8s-app=kube-proxy -o jsonpath="{.items[0].metadata.name}") -- ipvsadm -Ln

# Measure sync latency from kube-proxy metrics (if exposed)
kubectl get --raw /api/v1/nodes/worker-03/proxy/metrics |
  grep kubeproxy_sync_latency_seconds |
  tail -n 1

Root Cause Analysis

Node‑Local Rendering Lag and Stale Service State

The symptom arises when kube-proxy receives an EndpointSlice update but defers applying it until the next full resync cycle. During the intervening window, the node’s IPVS (or iptables) table still reflects the previous slice version, which may lack endpoints for a Pod that has just become Ready or has changed its IP due to rescheduling. Because the EndpointSlice object in etcd already shows the new endpoints, kubectl get endpointslices appears healthy, yet the dataplane is stale.

Impact of Caching and DNS Resolution

CoreDNS returns the stable ClusterIP; it does not participate in per‑node load‑balancing. DNS caching on the client side is irrelevant to the failure. The failure is purely dataplane‑local: the node’s connection tracking or IPVS scheduler forwards packets to the missing backend, resulting in a silent drop (no TCP RST, just a timeout).

Role of Load Balancers and Networking

If the service is exposed via an external load balancer (e.g., cloud LB with type: LoadBalancer), the LB forwards traffic to any node’s NodePort. The LB health checks may pass because at least one node has the full backend set, masking the per‑node hole. Traffic that lands on the affected node experiences the blackhole.

Resolving the Issue

Updating EndpointSlices and Service State

No direct update is needed; the EndpointSlice is already correct. The fix is to force kube-proxy to reconcile immediately:

# Restart kube-proxy on the affected node to trigger an immediate full sync
kubectl -n kube-system delete pod -l k8s-app=kube-proxy --field-selector spec.nodeName=worker-03

The pod is recreated by the DaemonSet; on startup it performs a full list/watch and programs IPVS with the current slice set.

Adjusting Node‑Local Rendering and Caching

Reduce the sync period to minimize the window of staleness:

# Edit the kube-proxy ConfigMap
kubectl -n kube-system edit configmap kube-proxy

Change:

kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-proxy
  namespace: kube-system
data:
  kubeproxy.conf: |
    ...
    syncPeriod: "5s"
    ...

Apply and roll the DaemonSet:

kubectl -n kube-system rollout restart daemonset kube-proxy

Monitor kubeproxy_sync_latency_seconds to ensure it stays below the new period.

Implementing Monitoring and Alerting

Alert on kubeproxy_sync_latency_seconds > syncPeriod/2.
Alert on mismatch between endpoint_slice_endpoints_total (from kube-state-metrics) and kube_ipvs_virtual_server_current_connections sum across backends.

Example PrometheusRule:

groups:
- name: kube-proxy.health
  rules:
  - alert: KubeProxySyncLag
    expr: avg_over_time(kubeproxy_sync_latency_seconds[5m]) > 10
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "kube-proxy sync lag on {{ $labels.instance }}"
      description: "Average sync latency exceeds 10s for the last 5 minutes."

Scaling Limitations and Considerations

Scaling EndpointSlices and Services

Each EndpointSlice holds ≤100 endpoints by default (--endpoint-slice-size). Large services (≥1000 Pods) generate many slices, increasing watch load on kube-proxy.
High slice count can exacerbate rendering lag because each slice triggers a separate diff computation.
Mitigation: increase --endpoint-slice-size (e.g., to 256) if the API server and etcd can handle larger objects.

Load Balancer and Networking Limitations

Cloud LB health checks often target the NodePort; they do not detect per‑node slice lag.
IPVS has a connection‑sync timeout (--ipvs-tcp-timeout); stale backends may cause connection resets if the timeout is low.
In pure iptables mode, latency is lower but the NAT rule rebuild is more CPU‑intensive on large slice counts.

Best Practices for Large‑Scale Deployments

Keep kube-proxy sync period ≤10 s for services with frequent churn.
Enable feature-gate=EndpointSliceProxying=true (default in v1.21+) to use the slice‑aware proxy path, which reduces the amount of data processed per sync.
Monitor kubeproxy_network_programming_latency_seconds and kubeproxy_iptables_rules_total.
Consider using IPVS with --ipvs-scheduler=lc (least‑connection) to better distribute load when slice updates are delayed.

Preventing Similar Failures

Implementing Health Checks and Monitoring

Deploy a DaemonSet that runs curl -sSf http://<svc-cluster-ip>:<port>/health from each node and exposes a Prometheus metric; alert on per‑node failure rate >0 %.
Use kube-state-metrics endpoint slice metrics to detect slices with notReady endpoints that persist beyond the pod’s terminationGracePeriodSeconds.

Regularly Updating and Verifying EndpointSlices

Add a CI step that runs:

kubectl get endpointslices -l kubernetes.io/service-name=<svc> -o jsonpath='{.items[*].endpoints[*].addresses}' | wc -w

and compares against:

kubectl get pods -l <selector> -o jsonpath='{.items[*].status.podIP}' | wc -w

Automate a weekly rollout restart of the kube-proxy DaemonSet during low‑traffic windows to clear any accumulated state.

Using Automation and CI/CD Pipelines

Integrate kubeconform or kubectl validation in PR pipelines to ensure Service and EndpointSlice manifests are syntactically correct.
Use GitOps (ArgoCD, Flux) to enforce that the kube-proxy ConfigMap is version‑controlled; any drift triggers a sync and redeploy.

Advanced Troubleshooting Techniques

Using `kubectl` and `kube-proxy` Logs

Enable