LinkState
Go back

Headless services, StatefulSets, and SANs that no longer match

Comparing Intended Peer Identity to Observed DNS Names and Presented Certificates in Headless StatefulSet Traffic

Introduction

A headless Service (clusterIP: None) does not allocate a virtual IP; it returns DNS A records that point directly to the pod IPs backing the Service. When paired with a StatefulSet, each replica gets a stable, ordinal‑based DNS name:

<statefulset-name>-<ordinal>.<service-name>.<namespace>.svc.cluster.local

This stable naming lets applications address peers by a deterministic identity that survives pod rescheduling, even though the underlying pod IP may change.

In mutual TLS (mTLS) or certificate‑based authentication, each pod must verify that the peer it talks to is the expected replica. Verification typically involves:

  1. Resolving the peer’s DNS name to an IP address.
  2. Performing a TLS handshake and checking the presented certificate’s Subject Alternative Name (SAN) against the expected identity.
  3. Applying any additional peer‑validation rules (hostname verification, certificate pinning, etc.).

A mismatch at any step aborts the TLS handshake, often appearing as transient connection errors that look like “pod flaps” in logs or monitoring, even though the pod remains Running.

Understanding Peer Identity and DNS Names

Intended Peer Identity

For a StatefulSet named web with three replicas, the intended peer identities are:

These names are stable across pod recreation; the ordinal never changes for a given pod unless the pod is deleted and a new pod with the same ordinal is created (which retains the name).

Observed DNS Names in Headless StatefulSets

A headless Service creates an A record for each pod:

web-0.web.default.svc.cluster.local. 30 IN  A 10.244.1.5
web-1.web.default.svc.cluster.local. 30 IN  A 10.244.1.6
web-2.web.default.svc.cluster.local. 30 IN  A 10.244.1.7

Clients that query the headless Service directly receive the current pod IP. If a pod is rescheduled, its IP changes but the A record is updated accordingly, so the observed DNS name matches the intended name as long as the resolver queries the headless Service directly.

Sources of Divergence

Divergence can occur when:

In such cases, the observed IP may correspond to a different ordinal, leading to a mismatch between the DNS name used for TLS verification and the certificate presented by the actual peer.

Presented Certificates in Headless StatefulSet Traffic

Certificate Presentation

During an mTLS handshake, the server presents its X.509 certificate. The client validates:

A common practice is to issue a certificate per ordinal, with SAN entries like:

DNS: web-0.web.default.svc.cluster.local
DNS: web-0.web

Alternatively, a wildcard SAN (*.web.default.svc.cluster.local) can be used for a single cert per StatefulSet.

Verification of Presented Certificates

Verification follows RFC 6125: the client attempts to match the hostname used in the connection (the DNS name resolved earlier) against each SAN entry. If the hostname is an IP address, the client checks the iPAddress SAN type. A mismatch aborts the TLS handshake with certificate_unknown or bad_certificate alert.

Impact of Certificate Mismatch

When the observed DNS name does not match any SAN in the presented certificate:

If the application treats TLS failures as transient and restarts the connection loop, the pod’s readiness probe may fail repeatedly, causing the pod to be marked NotReady and triggering a restart loop in some controllers.

Ordinal Moves and Peer Identity Verification

Ordinal Index in StatefulSets

Each pod in a StatefulSet has an immutable ordinal index assigned at creation time (0‑based). The ordinal is part of the pod’s hostname (<statefulset-name>-<ordinal>) and is used to generate the stable DNS name. The ordinal does not change when the pod is rescheduled; only if the pod is deleted and a new pod with the same ordinal is created does the ordinal persist.

Effect of Ordinal Moves on Peer Identity

Although the ordinal itself is static, the perceived ordinal can change if:

When the observed ordinal differs from the intended one, the DNS name used for TLS verification points to a different pod, whose certificate SAN set does not match, causing verification failure.

Troubleshooting Ordinal‑Related Peer Identity Issues

  1. Check the pod’s hostname: kubectl exec <pod> -- hostname should return <statefulset-name>-<ordinal>.
  2. Verify the StatefulSet’s ordinal assignment: kubectl get pods -l app=<app> -o custom-columns=NAME:.metadata.name,ORDINAL:.metadata.annotations.kubernetes\.io/pod-name.
  3. Inspect events for pod deletions/recreations: kubectl get events --field-selector involvedObject.name=<pod>.
  4. Ensure PVCs are correctly bound and that sufficient PVs exist for the requested storage class.
  5. If using external DNS, verify that it respects the headless Service’s A records and does not perform CNAME flattening that could change the ordinal portion.

Custom SAN Patterns and Peer Validation Rules

Custom Subject Alternative Name (SAN) Patterns

Operators may define SAN patterns that deviate from the default <statefulset>-<ordinal>.<service>.<namespace>.svc.cluster.local. Examples:

These patterns are specified in the certificate signing request (CSR) or via cert‑manager’s dnsNames field.

Peer Validation Rules in Headless StatefulSets

Beyond hostname matching, applications may enforce additional rules:

These rules are typically implemented in the TLS library configuration (e.g., Go’s tls.Config.VerifyPeerCertificate, Java’s X509TrustManager, or Envoy’s validation_context).

Configuring Custom SAN Patterns and Peer Validation Rules

Example using cert‑manager to issue a certificate with custom SANs for a StatefulSet named web:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: web-tls
  namespace: default
spec:
  secretName: web-tls-secret
  dnsNames:
    - web-0.web.default.svc.cluster.local
    - web-1.web.default.svc.cluster.local
    - web-2.web.default.svc.cluster.local
    - "*.web.default.svc.cluster.local"   # optional wildcard for future scale
  issuerRef:
    name: ca-issuer
    kind: Issuer

Application‑side verification (Go example):

tlsConfig := &tls.Config{
    VerifyPeerCertificate: func(rawCerts [][]byte, verifiedChains [][]*x509.Certificate) error {
        cert, err := x509.ParseCertificate(rawCerts[0])
        if err != nil {
            return err
        }
        hostname := "web-0.web.default.svc.cluster.local" // resolved earlier
        if err := cert.VerifyHostname(hostname); err != nil {
            return fmt.Errorf("hostname mismatch: %w", err)
        }
        // Additional pinning check
        expected := sha256.Sum256([]byte("pin-value"))
        actual := sha256.Sum256(cert.Raw)
        if !bytes.Equal(expected[:], actual[:]) {
            return errors.New("certificate pinning failed")
        }
        return nil
    },
}

Creating Failures that Resemble Random Pod Flaps

Simulating Random Pod Flaps with Ordinal Moves and Custom SAN Patterns

To reproduce the symptom:

  1. Deploy a headless StatefulSet with three replicas, each serving an mTLS endpoint.
  2. Issue certificates with SANs that only match the exact ordinal names (no wildcard).
  3. Introduce a scenario where a pod is deleted and its PVC cannot be re‑bound (e.g., insufficient storage). The StatefulSet controller will then create a new pod with the next ordinal (e.g., if pod web-1 is deleted and cannot reuse its PVC, a new pod web-3 may be created, leaving a gap).
  4. Configure clients to resolve peers via the headless Service but cache DNS responses for 30 s.
  5. After the ordinal gap appears, clients attempting to connect to web-1 will receive the IP of web-3 (due to DNS caching or external DNS load‑balancing). The TLS handshake fails because web-3’s certificate does not contain web-1 in its SAN.
  6. The client logs connection errors, retries, and the pod’s readiness probe fails, causing the pod to be marked NotReady. From an external viewpoint, the pod appears to flap.

Identifying and Troubleshooting Peer Identity Verification Failures

Key indicators:

Troubleshooting steps:

  1. Capture TLS handshake with tcpdump or openssl s_client -connect <pod-ip>:<port> -servername <expected-hostname>.
  2. Compare the presented certificate’s SAN list with the hostname used in the -servername flag.
  3. Verify DNS resolution at the moment of failure: kubectl exec <client-pod> -- nslookup web-0.web.default.svc.cluster.local.
  4. Check StatefulSet events for pod deletions/recreations: kubectl describe statefulset web.
  5. Ensure that the headless Service’s clusterIP is indeed None: kubectl get svc web -o yaml.

Code Examples for Reproducing and Debugging Peer Identity Issues

Reproducer (bash)

# 1. Create StorageClass with limited capacity (for demo)
cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: limited-sc
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
EOF

# 2. Create PVCs (only 2 PVs available)
for i in 0 1 2; do
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: web-pv-$i
  namespace: default
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 5Gi
  storageClassName: limited-sc
EOF
done

# 3. Deploy headless StatefulSet (3 replicas) with TLS
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  namespace: default
spec:
  serviceName: "web"
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: app
        image: gcr.io/google-samples/hello-app:1.0
        env:
        - name: PORT
          value: "8443"
        volumeMounts:
        - name: tls
          mountPath: /etc/tls
        - name: data
          mountPath: /data
      - name: envoy
        image: envoyproxy/envoy:v1.25-latest
        volumeMounts:
        - name: tls
          mountPath: /etc/envoy/tls
        args:
        - "-c"
        - "/etc/envoy/envoy.yaml"
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 5Gi
      storageClassName: limited-sc
  volumeMounts:
  - name: tls
    mountPath: /etc/tls
  - name: data
    mountPath: /data
EOF

# 4. Issue certs with exact ordinal SANs (using cert-manager)
# (Assume cert-manager and issuer already installed)
for i in 0 1 2; do
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: web-tls-$i
  namespace: default
spec:
  secretName: web-tls-secret-$i
  dnsNames:
    - web-${i}.web.default.svc.cluster.local
  issuerRef:
    name: ca-issuer
    kind: Issuer
EOF
done

Debugging Commands

# Check pod hostname
kubectl exec web-0 -- hostname

# Verify ordinal assignment
kubectl get pods -l app=web -o custom-columns=NAME:.metadata.name,ORDINAL:.metadata.annotations.kubernetes\.io/pod-name

# Look for deletion/recreation events
kubectl get events --field-selector involvedObject.name=web-0

# Test TLS handshake with expected hostname
openssl s_client -connect 10.244.1.6:8443 -servername web-1.web.default.svc.cluster.local -showcerts

# Verify DNS resolution at runtime
kubectl exec <client-pod> -- nslookup web-0.web.default.svc.cluster.local

These steps help isolate whether the failure stems from ordinal drift, DNS caching, or SAN mismatches, allowing you to correct the underlying configuration rather than treating the symptom as random pod flapping.

Share this post on:
Share this post via WhatsApp Share this post on Facebook Share this post on X Share this post via Telegram Share this post on Pinterest Share this post via email
Previous Post
Leases, quorums, and fencing for source of truth
Next Post
Which drops were qdisc and which were memory