Double TLS at the egress gateway boundary

Introduction to Sidecar Encryption and Egress Gateway

Sidecar encryption in a service mesh means each pod runs a sidecar proxy (Envoy, Linkerd, etc.) that terminates mutual TLS (mTLS) from the client and re‑encrypts traffic to the next hop. The proxy performs the TLS handshake, validates peer certificates, and forwards clear‑text (or re‑encrypted) data to the local application or another proxy. Cryptographic work happens in user space; the kernel sees only TCP segments after TLS decryption/re‑encryption.

An Egress Gateway is a dedicated set of proxy instances at the mesh boundary that originates TLS to services outside the mesh. Unlike the sidecar, which only encrypts traffic to the mesh, the egress gateway terminates inbound mTLS from the sidecar, applies policy checks (authorization, SNI matching, header manipulation), and initiates a new TLS connection to the external endpoint. This separation lets you enforce egress‑specific policies (rate limits, TLS credentials) without touching every sidecar.

Request Flow Through Sidecar Encryption

Initial Request and Sidecar Encryption Process

Consider an HTTP GET from client-app to https://api.example.com.

1. The application writes the HTTP request to a local Unix domain socket or localhost TCP port that the sidecar listens on (e.g., 127.0.0.1:15001 for Istio’s inbound listener).
2. The sidecar receives the TCP segment, performs TLS decryption using the inbound mTLS context (client‑sidecar), and passes the plaintext HTTP to the outbound filter chain.
3. The outbound filter chain applies routing (VirtualService) and forwards to the egress gateway subset (egressgateway.istio-system.svc:80).
4. The sidecar encrypts the outbound data using the outbound mTLS context (sidecar‑to‑egressgateway) and writes to the socket bound to the egress gateway’s VIP.

At each step the proxy:

• Receives data via recvfrom() on a non‑blocking fd (kernel → user space).
• Decrypts/encrypts with OpenSSL/BoringSSL (async path).
• Sends via sendmsg() to the next hop’s socket (kernel TX ring).

If the sidecar is misconfigured to double encrypt (both inbound and outbound TLS enabled for the same destination), the plaintext after the first decryption is re‑encrypted, producing TLS‑inside‑TLS on the wire. The external server sees garbage, aborts the TLS handshake, and the connection appears as “connection reset by peer”.

Example Configuration for Sidecar Encryption

# Enable STRICT mTLS for the namespace (forces inbound & outbound encryption)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: bookinfo
spec:
  mtls:
    mode: STRICT

To encrypt only inbound traffic and leave outbound plaintext for the egress gateway:

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: bookinfo
spec:
  mtls:
    mode: PERMISSIVE   # inbound mTLS required, outbound optional

Then enforce mTLS from sidecar to egress gateway via a DestinationRule:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: egressgateway
  namespace: istio-system
spec:
  host: egressgateway.istio-system.svc
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL   # sidecar → egressgateway mTLS

CLI Commands for Verifying Sidecar Encryption

# 1. Check inbound mTLS listener (15006) on the sidecar
istioctl proxy-config listeners <pod-name> -n bookinfo | grep 15006

# 2. Verify outbound routes to egressgateway
istioctl proxy-config routes <pod-name> -n bookinfo | grep egressgateway

# 3. Capture TLS handshake on the sidecar’s listener port
kubectl exec -n bookinfo <pod-name> -- tcpdump -nn -s0 -i any port 15001 -w /tmp/sidecar.pkt

# 4. (Lab only) Decrypt capture with the sidecar’s key using Wireshark or sslsplit
#    Never expose production keys.

Egress Gateway and SNI Handling

How Egress Gateway Handles SNI

The egress gateway receives plain‑text HTTP/HTTP/2 from the sidecar. Before originating an outbound TLS connection it must decide the hostname for the TLS ClientHello SNI field. The decision hierarchy is:

1. host attribute of the ServiceEntry or VirtualService routing to the external service.
2. Explicit sni field in the DestinationRule for the external host.
3. Fallback to the authority header (:authority for HTTP/2, Host for HTTP/1.1) if no SNI is configured.

Mismatched SNI causes the external server to abort the handshake (certificate_unknown/bad_certificate), and the gateway returns a 502/503.

Configuring SNI Handling in Egress Gateway

Control SNI via a DestinationRule that targets the external hostname (defined in a ServiceEntry):

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: external-api
  namespace: istio-system
spec:
  host: api.example.com   # must match ServiceEntry host
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    tls:
      mode: ISTIO_MUTUAL   # optional, if external service also does mTLS
      sni: api.example.com # explicit SNI override

To copy the incoming Host header into SNI, use an EnvoyFilter that sets dynamic metadata:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: sni-from-header
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: egressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.lua
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
          inlineCode: |
            function envoy_on_request(request_handle)
              local host = request_handle:headers():get(":authority")
              if host then
                request_handle:dynamicMetadata():setKey("envoy.lua", "sni", host)
              end
            end

Reference that metadata in the DestinationRule using the placeholder %DYNAMIC_METADATA(envoy.lua:sni)% (Istio 1.19+).

Example Configuration for an External HTTPS Service

# 1. Declare the external service
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-api
  namespace: istio-system
spec:
  hosts:
  - api.example.com
  location: MESH_EXTERNAL
  resolution: DNS
  ports:
  - number: 443
    name: https
    protocol: TLS

# 2. Route to the egress gateway
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: external-api
  namespace: istio-system
spec:
  hosts:
  - api.example.com
  gateways:
  - mesh
  - istio-system/egressgateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: api.example.com
        subset: egressgateway

# 3. Define the egress gateway subset (sidecar → egressgateway mTLS)
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: egressgateway
  namespace: istio-system
spec:
  host: egressgateway.istio-system.svc
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    tls:
      mode: ISTIO_MUTUAL

# 4. External TLS (SNI) handling
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: external-api-tls
  namespace: istio-system
spec:
  host: api.example.com
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL   # set to DISABLE if external does not do mTLS
      sni: api.example.com

Upstream Validation and Proxy Chain

Overview of Upstream Validation

After the egress gateway establishes the outbound TLS connection, it performs upstream validation before forwarding application data:

• Certificate verification – builds chain to a trusted root; checks hostname against SNI or subjectAltName.
• Certificate revocation – OCSP stapling or CRL checks (if enabled).
• Authorization policies – e.g., AuthorizationPolicy that checks source.principal (the sidecar’s identity) against allowed actions on the external host.
• Protocol constraints – enforce TLS 1.2+, disable weak ciphers, require specific ALPN (e.g., h2 for HTTP/2).

Failure results in a 502 (Bad Gateway) with details in the access log (%REQ(:authority)%, %UPSTREAM_TRANSPORT_FAILURE_REASON%).

Configuring Upstream Validation

In Istio, upstream TLS validation is controlled by the DestinationRule.tls block:

trafficPolicy:
  tls:
    mode: ISTIO_MUTUAL   # or SIMPLE for one-way TLS
    sni: api.example.com
    # Trust a specific root CA
    caCertificates: |
      -----BEGIN CERTIFICATE-----
      ... (root CA) ...
      -----END CERTIFICATE-----
    # Enforce TLS version range
    tlsMinimumProtocolVersion: TLSv1_2
    tlsMaximumProtocolVersion: TLSv1_3
    # Whitelist cipher suites (OpenSSL format)
    cipherSuites:
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Attach an AuthorizationPolicy to the egress gateway workload to validate the caller’s identity:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: egress-external-api
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: egressgateway
  action: ALLOW
  rules:
  - to:
    - host:
      - api.example.com
    when:
    - key: source.principal
      values: ["cluster.local/ns/bookinfo/sa/client-app-sa"]

Example Code for Upstream Validation Configuration

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: external-api-validation
  namespace: istio-system
spec:
  host: api.example.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    tls:
      mode: ISTIO_MUTUAL
      sni: api.example.com
      caCertificates: |
        -----BEGIN CERTIFICATE-----
        ... (root CA) ...
        -----END CERTIFICATE-----
      tlsMinimumProtocolVersion: TLSv1_2
      tlsMaximumProtocolVersion: TLSv1_3
      cipherSuites:
      - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

This walkthrough shows how a miswired proxy chain—double TLS, incorrect SNI, or failed upstream validation—can make a healthy external service appear down, and how to configure and verify each step in an Istio‑based service mesh.