Root CA rotation partitioned one tenant after a seemingly clean rollout

Incident Overview

Incident Summary

At 02:14 UTC on 2025‑08‑12, the tenant‑isolated micro‑service group svc‑frontend‑a began returning TLS handshake failures (SSL_alert_handshake_failure) to external clients, while all other service groups in the same tenant continued to serve traffic successfully. Dashboards for the automated root‑CA rotation pipeline showed a green “rotation complete” status at 01:58 UTC, and the new root CA (root‑ca‑2025‑08) was present in the central trust‑anchor repository. Investigation revealed that the sidecar proxies attached to svc‑frontend‑a were still using the stale root CA (root‑ca‑2024‑11) because the sidecar’s trust‑anchor file had not been reloaded after rotation, creating a trust‑anchor skew that isolated only that service group.

Incident Timeline

Affected Services

• Tenant: acme‑corp‑tenant
• Namespace: acme‑corp‑tenant‑prod
• Impacted service group: svc‑frontend‑a (3 replicas, each with Istio sidecar istio-proxy)
• Unaffected service groups: svc‑backend‑b, svc‑api‑c, svc‑worker‑d (same sidecar image, different pod restart schedules allowed them to pick up the new trust bundle earlier).

Root Cause Analysis

Initial Assessment

The initial hypothesis was a silent failure of the CA rotation job because the dashboard showed success while the service exhibited TLS failures. Verification of the central ConfigMap and Secret confirmed the presence of the new root CA (root‑ca‑2025‑08). The discrepancy pointed to a failure in the consumption of the updated trust anchor by the sidecar proxies.

Deep Dive Analysis

Root CA Rotation Process

1. Generation – HashiCorp Vault PKI role creates a new self‑signed root CA (root‑ca‑2025‑08) with a 10‑year validity.
2. Signing – The new root signs the existing intermediate CA (intermediate‑ca‑2023‑04) to preserve existing leaf certificates.
3. Distribution – Concatenated PEM bundle (root‑ca‑2025‑08.pem + intermediate‑ca‑2023‑04.pem) is written to:
   • ConfigMap trust-anchors (used by admission controllers for validating webhook TLS).
   • Secret sidecar‑tls‑secrets (mounted at /etc/istio/certs/root-ca.pem in each sidecar container).
4. Notification – The job emits a Prometheus metric ca_rotation_success{status="1"} and writes a completion entry to an audit log.

All steps completed without error; the metric showed 1 at 01:58 UTC.

Sidecar Configuration

• Image: docker.io/istio/proxyv2:1.18.0
• Mount: sidecar‑tls‑secrets mounted as read‑only volume at /etc/istio/certs.
• Reload mechanism: The sidecar process relies on the container’s PID 1 (istio-proxy) to receive SIGHUP from the Istio agent (istiod) when the mounted secret changes. The agent uses Kubernetes’ watch API to detect secret updates and then sends SIGHUP to all matching pods.
• Observed failure: The istiod agent successfully detected the secret update and sent SIGHUP to the pod, but the sidecar did not reload the volume because the underlying inotify watch on the mounted file descriptor had been silenced by a prior SIGSYS seccomp filter that blocked inotify_add_watch for the sidecar’s UID. Consequently, the sidecar continued to read the stale PEM file from its memory‑mapped cache.

Trust Anchor Management

• Trust anchor source: Sidecars read the PEM file at process start and cache the parsed X.509 trust store in memory.
• Expected reload: Upon SIGHUP, the sidecar re‑opens the file, re‑parses the PEM, and replaces the cached trust store.
• Actual behavior: Because the file descriptor watch was ineffective, the sidecar never received the notification to re‑open the file; the cached trust store remained anchored to the old root CA (root‑ca‑2024‑11). When presenting its own leaf certificate (signed by the new intermediate), the sidecar could not verify the peer’s certificate chain, leading to the TLS alert.

Thus, the root cause is the sidecar’s inability to reload its trust‑anchor volume due to a blocked inotify watch, causing a trust‑anchor skew that affected only the subset of pods that had not been restarted since the seccomp profile was applied.

Amplifying Factors

Inadequate Monitoring

• No metric existed to compare the trust‑anchor hash in the sidecar’s mounted volume with the hash of the secret in the API server.
• The existing ca_rotation_success metric only reflected job completion, not propagation to data‑plane agents.
• Sidecar logs were not aggregated into a central system with real‑time parsing for failed to verify peer certificate patterns.

Insufficient Logging

• The sidecar’s SIGHUP handler logged only at debug level, which was not retained in the default log sink (log level set to info).
• The seccomp audit log (audit.log) that would have shown blocked inotify_add_watch syscalls was not shipped to the observability pipeline.
• Consequently, the engineer had to manually kubectl exec into a pod and run strace to discover the blocked syscall.

Incomplete Automation

• The automation pipeline assumed that a secret update would trigger a sidecar reload via the Istio agent; it did not validate that the sidecar actually reloaded its trust bundle.
• No post‑rotation verification step (e.g., a curl --cacert check from a test pod) was included in the pipeline’s canary stage.
• The seccomp profile that blocked inotify was applied via a Helm chart update three weeks prior, but the chart’s release notes did not mention the impact on trust‑anchor reloads, and no regression test was added.

Detection Gaps

Limitations of Current Tooling

• Prometheus alerts were based solely on job success and service‑level SLOs; there was no alert for “trust-anchor mismatch between secret and sidecar volume”.
• Log‑based detection relied on error strings appearing in application logs; the TLS handshake failure was occurring inside the sidecar before traffic reached the application, so no application‑level error was logged.
• Configuration drift detection tools (e.g., kube-bench, opa) were not configured to check the seccompProfile field for syscalls relevant to TLS trust‑anchor reloads.

Inadequate Alerting

• The PagerDuty alert that fired was a generic “high error rate” alert; it did not contain contextual data pointing to a TLS trust issue.
• No alert existed for the metric istio_sidecar_tls_handshake_failure_total because the sidecar does not expose this metric by default; it would have required enabling proxy.stats matching.

Lack of Anomaly Detection

• No statistical baseline existed for the rate of SSL_alert_handshake_failure per service group; a sudden increase in one group was not flagged as anomalous because the overall tenant error rate remained below the threshold due to other healthy groups.
• The trust‑anchor version was not treated as a dimension in the metrics pipeline, preventing correlation of a version mismatch with error spikes.

Troubleshooting Steps

Initial Troubleshooting

1. Verified the CA rotation job’s output logs – showed success.
2. Checked the ConfigMap trust-anchors and Secret sidecar‑tls‑secrets – both contained the new root CA PEM.
3. Executed openssl x509 -in /etc/istio/certs/root-ca.pem -text -noout inside a sidecar container – returned the old root CA (root‑ca‑2024‑11).
4. Reviewed Istio agent logs (istiod) – confirmed that a secret update event was published and SIGHUP sent to the pod at 02:02 UTC.

In‑Depth Troubleshooting

CLI Commands Used

# 1. Verify secret contents
kubectl -n acme-corp-tenant-prod get secret sidecar-tls-secrets -o jsonpath='{.data.root-ca\.pem}' | base64 -d | openssl x509 -noout -subject -dates

# 2. Check sidecar volume mount
kubectl -n acme-corp-tenant-prod exec deploy/svc-frontend-a -c istio-proxy -- ls -l /etc/istio/certs/root-ca.pem

# 3. Examine inotify watch status (requires strace)
kubectl -n acme-corp-tenant-prod exec deploy/svc-frontend-a -c istio-proxy -- strace -e trace=inotify_add_watch,inotify_rm_watch -p $(pgrep istio-proxy) 2>&1 | head -20

# 4. Review seccomp profile applied to the pod
kubectl -n acme-corp-tenant-prod get pod <pod-name> -o jsonpath='{.spec.securityContext.seccompProfile}'
# Output showed:
#   {
#     "type": "Localhost",
#     "localhostProfile": "profiles/seccomp/strict.json"
#   }
# The strict profile blocked `inotify_add_watch` for non‑root UIDs.

# 5. Force sidecar reload via restart
kubectl -n acme-corp-tenant-prod rollout restart deployment/svc-frontend-a

Code Snippets Analyzed

Istio agent secret watcher (simplified):

func (s *secretWatcher) Run(stopCh <-chan struct{}) {
    informer := s.kube.InformerFactory.Core().V1().Secrets().Informer()
    informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
        UpdateFunc: func(old, new interface{}) {
            if !reflect.DeepEqual(old.(*v1.Secret).Data, new.(*v1.Secret).Data) {
                s.podLister.Pods(s.namespace).List(labels.Everything(), func(p *v1.Pod) error {
                    if podHasSidecar(p) {
                        s.signalPod(p) // sends SIGHUP
                    }
                    return nil
                })
            }
        }
    })
    informer.Run(stopCh)
}

The watcher correctly detected the secret change and signaled the pod. The failure point was not in this code.

Sidecar SIGHUP handler (from proxy source):

static void
reload_cert_chain(int sig) {
    if (sig == SIGHUP) {
        TRACE("reload: SIGHUP received");
        if (load_root_certs("/etc/istio/certs/root-ca.pem") != 0) {
            ERROR("failed to reload root certs");
        }
    }
}

The function is called, but load_root_certs() reads from a memory‑mapped copy of the file that was never unmapped because the file descriptor watch never triggered a munmap/mmap cycle. The seccomp block prevented the inotify_add_watch syscall that would have notified the sidecar’s runtime library (libevent) to invalidate the mapping.

Code and CLI Examples

Relevant Configuration Files

values.yaml (Helm) – seccomp profile:

podSecurityContext:
  seccompProfile:
    type: Localhost
    localhostProfile: profiles/seccomp/strict.json

profiles/seccomp/strict.json (excerpt):

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "syscalls": [
    { "name": "inotify_add_watch", "action": "SCMP_ACT_ERRNO" },
    { "name": "inotify_rm_watch",  "action": "SCMP_ACT_ERRNO" }
  ]
}

Istio sidecar deployment snippet:

containers:
- name: istio-proxy
  image: docker.io/istio/proxyv2:1.18.0
  volumeMounts:
  - name: tls-secrets
    mountPath: /etc/istio/certs
    readOnly: true
volumes:
- name: tls-secrets
  secret:
    secretName: sidecar-tls-secrets

CLI Commands for Root CA Rotation

# Example: generate new root CA and push to cluster (simplified)
vault pki issue -domain="internal.acme-corp" -ttl=8760h > new-ca.pem
kubectl -n acme-corp-tenant-prod create configmap trust-anchors --from-file=root-ca.pem=new-ca.pem --dry-run=client -o yaml | kubectl apply -f -
kubectl -n acme-corp-tenant-prod create secret generic sidecar-tls-secrets --from-file=root-ca.pem=new-ca.pem --dry-run=client -o yaml | kubectl apply -f -