Session affinity intent versus real backend stickiness

Service Session Affinity: Declaration vs. Observed Stickiness

Session affinity (sticky sessions) directs packets from the same client source to a single backend endpoint for the duration of a session. In Kubernetes the affinity is declared on a Service (spec.sessionAffinity: ClientIP or None). The control plane translates that declaration into dataplane state that kube‑proxy (or a CNI‑based proxy) installs on each node.

Stateful workloads—HTTP apps with in‑memory caches, WebSocket servers, databases using connection‑pooling—rely on a client staying on one backend. Without affinity, subsequent requests may hit a different replica, causing cache misses, re‑authentication, or protocol errors. Operators configure affinity to reduce latency and avoid application‑level session replication.

Declared Service Session Affinity

Configuration Options

The affinity timeout (spec.sessionAffinityConfig.clientIP.timeoutSeconds) controls how long the hash mapping lives after the last seen packet; default = 10800 s (3 h). Changing the timeout only affects garbage‑collection, not the hash algorithm.

Example Service

apiVersion: v1
kind: Service
metadata:
  name: sticky-svc
spec:
  selector:
    app: myapp
  ports:
  - port: 80
    targetPort: 8080
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 1800   # 30 min

When the Service is created, the control plane writes the declaration to etcd. The node‑local proxy (kube‑proxy or a CNI implementation) watches the Service and renders the affinity into its dataplane.

Observed Backend Stickiness

iptables

Configuration

In iptables mode kube‑proxy creates a NAT chain (KUBE-SERVICES) that jumps to a service‑specific chain (KUBE-SVC-<hash>). Each endpoint gets a rule that performs DNAT to the pod IP.

• With sessionAffinity: None kube‑proxy inserts a statistic match (--mode nth --every 1 --packet 0).
• With ClientIP it replaces the nth‑match with a recent match that tracks the source IP:

-A KUBE-SVC-XYZ -m recent --name KUBE-SEP-ABCD --rsource --seconds 1800 --reap \
    -j DNAT --to-destination 10.1.2.3:8080

The --recent match maintains a per‑source‑IP list; the first packet creates an entry, subsequent packets within the timeout hit the same entry and are DNAT‑ed to the same backend.

Example Rules

$ iptables -t nat -L KUBE-SVC-XYZ -v -n
Chain KUBE-SVC-XYZ (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            recent: name KUBE-SEP-ABCD side/source mask 255.255.255.255 seconds 1800 hitcount 1 rsource --rsource -j DNAT --to-destination 10.1.2.3:8080
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            recent: name KUBE-SEP-EFGH side/source mask 255.255.255.255 seconds 1800 hitcount 1 rsource --rsource -j DNAT --to-destination 10.1.2.4:8080

The first matching rule (based on endpoint order) wins for a given source IP; the recent list ensures later packets from that IP hit the same rule.

Troubleshooting

Common pitfalls

• Connection‑tracking timeouts – The recent match tracks packets only. If the underlying conntrack entry expires (default TCP 5 min, UDP 30 s) while the recent entry is still valid, subsequent packets may create a new conntrack tuple and bypass the recent match, causing apparent affinity loss.
• Endpoint churn – When an endpoint is removed, its recent list entry is not flushed automatically; stale entries can DNAT packets to a non‑existent pod, resulting in connection resets.
• NAT masquerade interference – If the node performs SNAT for outbound traffic (e.g., MASQUERADE in POSTROUTING) before the PREROUTING DNAT, the source IP seen by the recent match may be the node IP, breaking per‑client stickiness.

Debugging tools

• iptables -t nat -L -v -n – view packet/byte counters per rule; increasing counters on a DNAT rule indicate hits.
• iptables -t nat -L KUBE-SEP-<hash> -v -n – inspect the endpoint‑specific chain that holds the recent match.
• conntrack -L -p tcp --dport 8080 – verify that conntrack entries survive as long as the recent entry.
• iptables -t nat -Z – zero counters to measure fresh traffic.
• tcpdump -i any -nn port 8080 – correlate client IP with backend IP observed in packets.

IPVS

Configuration

In IPVS mode kube‑proxy creates a virtual server (vs) for each Service IP:port. The scheduler determines packet distribution.

• For ClientIP affinity IPVS uses the sh (source hash) scheduler, which hashes the source IP (and optionally source port) to pick a real server.
• Key parameters:

Example Commands

# Show the virtual server
$ ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.10:80 sh
  -> 10.1.2.3:8080               Masq    1      0          0
  -> 10.1.2.4:8080               Masq    1      0          0

If sessionAffinityConfig.clientIP.timeoutSeconds: 1800 is set, kube‑proxy adds a persistence timeout:

$ ipvsadm -Ln --timeout
TCP  10.96.0.10:80 sh persistent_timeout 1800

Troubleshooting

Limitations

• Persistence vs. connection tracking – persistent_timeout applies only to new packets after a connection ends. Long‑lived TCP connections (e.g., HTTP/2) stay bound to the initial real server regardless of the persistence timeout; if the backend pod dies, the connection breaks and a new connection may be load‑balanced again, causing a perceived affinity loss.
• Hash collisions – With few real servers, multiple client IPs may hash to the same server (expected). Scaling the server set up or down can shift the hash mapping for many clients, causing a sudden reshuffle.
• IPVS sync daemon – In clusters with multiple nodes running kube‑proxy in IPVS mode, the ipvsadm --save/--restore mechanism keeps IPVS tables consistent. Lag in the sync daemon can leave some nodes with outdated real‑server lists, leading to mismatched affinity.

Debugging techniques

• ipvsadm -Ln – view virtual server, scheduler, and real‑server stats.
• ipvsadm -Ln --stats – show active/inactive connection counts per real server.
• ipvsadm -Ln --timeout – verify persistence timeout values.
• cat /proc/net/ip_vs_stats – overall IPVS packet counters.
• ipvsadm -Lnc – show connection table entries (source IP → real server) to confirm persistence.
• ss -tnp state ESTABLISHed '( dport = :80 )' – cross‑check which backend a given client IP is connected to.

eBPF

Configuration

In eBPF mode (used by kube‑proxy’s experimental eBPF dataplane or CNIs such as Cilium) a Service is represented as an eBPF hash map (service_map) that maps a service key (cluster IP + port + protocol) to a backend map containing an array of endpoint IDs. Affinity is implemented by a second map (affinity_map) that hashes the source IP (and optionally source port) to an index into the backend array. The map is updated when endpoints change; existing entries remain until they expire (based on a TTL stored in the map value) or are overwritten.

Typical eBPF program flow:

1. Packet enters the XDP or tc ingress hook.
2. Lookup service key → backend array.
3. If sessionAffinity: ClientIP, compute hash(source IP) % N → index.
4. Read backend IP/port from the array at that index; perform bpf_redirect or bpf_sk_redirect_map to the backend socket.
5. Update/write affinity map entry with a timestamp; periodic garbage‑collection removes stale entries.

Example eBPF Maps (C‑like pseudocode)

/* Service → backend array ID */
struct bpf_map_def SEC("maps") service_map = {
    .type = BPF_MAP_TYPE_HASH,
    .key_size = sizeof(struct svc_key),
    .value_size = sizeof(__u32),   // backend_array_id
    .max_entries = 65535,
};

/* Backend array of arrays: [ [IP,port], [IP,port], ... ] */
struct bpf_map_def SEC("maps") backend_array = {
    .type = BPF_MAP_TYPE_ARRAY_OF_ARRAYS,
    .key_size = sizeof(__u32),     // backend_array_id
    .value_size = sizeof(__u64) * 2, // [IP, port] per slot
    .max_entries = 256,
};

/* Affinity map: source IP → {backend_index, timestamp} */
struct bpf_map_def SEC("maps") affinity_map = {
    .type = BPF_MAP_TYPE_HASH,
    .key_size = sizeof(__u32),     // source IP (IPv4)
    .value_size = sizeof(__u64),   // (index<<32) | timestamp
    .max_entries = 131072,
};

/* XDP entry point */
SEC("xdp")
int xdp_sticky(struct xdp_md *ctx)
{
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;
    struct ethhdr *eth = data;
    if (data + sizeof(*eth) > data_end) return XDP_PASS;
    if (eth->h_proto != htons(ETH_P_IP)) return XDP_PASS;

    struct iphdr *ip = data + sizeof(*eth);
    if (data + sizeof(*eth) + sizeof(*ip) > data_end) return XDP_PASS;

    struct svc_key key = {
        .ip   = ip->daddr,
        .port = ((struct tcphdr *)(data + sizeof(*eth) + sizeof(*ip)))->dest,
        .proto = ip->protocol,
    };
    __u32 *backend_id = bpf_map_lookup_elem(&service_map, &key);
    if (!backend_id) return XDP_PASS;

    __u32 idx;
    if (key.proto == IPPROTO_TCP) {
        /* ClientIP affinity */
        __u32 hash = bpf_hash(&ip->saddr, 4, 0);
        idx = hash % BACKEND_COUNT(*backend_id);
        /* Update affinity map with current time */
        __u64 val = ((__u64)idx << 32) | bpf_ktime_get_ns();
        bpf_map_update_elem(&affinity_map, &ip->saddr, &val, BPF_ANY);
    } else {
        /* UDP or other protocols: fall back to round‑robin */
        idx = 0; /* placeholder – actual implementation may use a counter */
    }

    __u64 *backend = bpf_map_lookup_elem(&backend_array, backend_id);
    if (!backend) return XDP_PASS;

    __u64 slot = backend[idx * 2];   // IP
    __u64 port = backend[idx * 2 + 1]; // port
    struct iphdr *new_ip = data + sizeof(*eth);
    new_ip->daddr = slot;
    struct tcphdr *new_tcp = data + sizeof(*eth) + sizeof(*iphdr);
    new_tcp->dest = port;

    return XDP_TX;
}

The snippet above illustrates the core lookup, hash‑based index selection, affinity‑map update, and packet rewrite. Production implementations include additional safety checks, handling of IPv6, and fallback paths.

Troubleshooting

Common issues

• Map exhaustion – If the affinity_map reaches max_entries, further updates fail with -ENOSPC, causing packets to bypass affinity. Monitor with bpftool map show.
• Stale entries – Entries are removed only by a periodic garbage‑collector; if the interval is too long, affinity may persist beyond the intended timeout. Adjust the TTL stored in the map value or the collector frequency.
• Verification mismatch – The hash used in the eBPF program must match the one expected by the operator (e.g., inclusion of source port). A mismatch leads to apparent stickiness failures.

Debugging techniques

• bpftool map show name affinity_map – dump current affinity entries (source IP → index/timestamp).
• bpftool prog show – verify the XDP/tc program is attached to the correct interface.
• tc -s filter show dev <iface> ingress – see packet counters for the eBPF filter.
• `tcpdump -i any -nn port