Minimum telemetry to prove the mesh is the outage

Introduction

When a service reports an outage, the first operational question is: Is the failure inside the Envoy sidecar (including the mTLS handshake) or inside the application container? Answering this quickly reduces mean‑time‑to‑identify (MTTI) and prevents wasted effort on the wrong tier. Telemetry that is generated as close to the failure point as possible—Envoy’s access logs, counters, and TLS‑freshness signals—provides the evidence needed to prove or disprove an Envoy‑ or mTLS‑path origin without relying on application‑level instrumentation that may be missing during an outage.

Access‑Log Fields for Telemetry

HTTP Request and Response Fields

Envoy’s access log can be formatted to emit request‑level details that directly map to the request lifecycle:

TLS Handshake and Certificate Fields

When mTLS is enabled, Envoy can add TLS‑specific fields to the same access‑log line:

Example Access‑Log Configuration for Envoy

A minimal, production‑ready configuration that captures the fields above while keeping cardinality manageable (no per‑request headers or raw bodies). The log is written to /var/log/envoy/access.log and streamed via gRPC to a collector (e.g., Fluent Bit) for real‑time querying.

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 10000 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route: { cluster: service_backend }
          http_filters:
          - name: envoy.filters.http.router
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: /var/log/envoy/access.log
              format: |
                [%START_TIME%] "%REQ(:method)% %REQ(:path)% %PROTOCOL%" \
                %RESPONSE_CODE% %RESPONSE_FLAGS% %DURATION% \
                %UPSTREAM_HOST% %UPSTREAM_TRANSPORT_FAILURE_REASON% \
                %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% \
                %DOWNSTREAM_TLS_SESSION_ID% %DOWNSTREAM_TLS_CIPHER% %DOWNSTREAM_TLS_SNI% \
                %DOWNSTREAM_LOCAL_CERTIFICATE% %DOWNSTREAM_PEER_CERTIFICATE% \
                %DOWNSTREAM_PEER_CERT_SUBJECT% %DOWNSTREAM_PEER_CERT_ISSUER% \
                %DOWNSTREAM_PEER_CERT_SERIAL% %DOWNSTREAM_PEER_CERT_NOT_BEFORE% %DOWNSTREAM_PEER_CERT_NOT_AFTER% \
                %DOWNSTREAM_TLS_ERROR%
          - name: envoy.access_loggers.http_grpc
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.GenericAccessLog
              transport_api_version: V3
              common_config:
                name: envoy_grpc_access_log
              grpc_service:
                envoy_grpc:
                  cluster_name: accesslog_cluster

Note – Envoy does not emit a metric for certificate expiration directly; the expiration must be derived from %DOWNSTREAM_PEER_CERT_NOT_AFTER% (or from SDS). If the access‑log format omits the TLS fields, the mTLS path becomes invisible.

Counter Families for Outage Detection

Request and Response Counters

Envoy exports a hierarchical set of counters under the stats sink (Prometheus, StatsD, etc.). The most relevant families for distinguishing Envoy vs. app failures are:

Error and Failure Counters

Beyond the request families, Envoy provides error counters that surface internal processing problems:

Example Counter Configuration for Envoy

Envoy exposes these counters automatically when a stats_sink is configured. A minimal Prometheus scrape setup using the StatsD sink (replace with Prometheus exporter as needed):

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }

stats_sinks:
- name: envoy.metrics
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.statsd.v3.StatsdSink
    tcp_cluster_name: statsd_cluster
    prefix: envoy

If Prometheus is used via the envoy-prometheus exporter, the same counters appear under the envoy namespace.

Note – Envoy does not expose a counter for “certificate expired” or “certificate about to expire”. Those must be derived from the access‑log TLS fields or from an external secret‑management health check.

Certificate Freshness Signals

Certificate Expiration and Renewal Monitoring

The mTLS path is healthy only if both sides present a valid, non‑expired certificate. Envoy provides the peer certificate’s notAfter timestamp via the access‑log field %DOWNSTREAM_PEER_CERT_NOT_AFTER%. To turn this into a usable signal:

1. Parse the ISO‑8601 timestamp from each access‑log line (or from a periodic SDS dump).
2. Compute time_to_expiry = notAfter - now.
3. Emit a gauge (e.g., mtls_peer_cert_time_to_expiry_seconds) to a monitoring system.

If the gauge drops below a threshold (e.g., 86 400 s for 1 day), an alert fires indicating an impending cert expiry.

Certificate Validation and Verification Signals

Beyond expiration, validation failures are captured by:

• %DOWNSTREAM_TLS_ERROR% – contains strings like certificate_verify_failed, unsupported_certificate, certificate_required.
• tls_handshake_failed counter – increments when the TLS handshake aborts for any reason, including cert validation.
• ssl_connection_error_total{reason="certificate_unknown"} – fine‑grained error reason if the listener is configured with ssl stats.

Example Certificate Freshness Signal Configuration

Assuming a sidecar Fluent Bit agent tails Envoy’s access log and emits metrics to Prometheus via the prometheus_exporter output:

[INPUT]
    Name tail
    Path /var/log/envoy/access.log
    Parser envoy_access_log
    Tag envoy.access

[PARSER]
    Name   envoy_access_log
    Format regex
    Regex  ^\[(?<start_time>[^\]]+)\] "(?<method>[^ ]+) (?<path>[^ ]+) (?<proto>[^ ]+)" (?<status>\d+) (?<flags>\S+) (?<duration>\d+) (?<upstream_host>\S+) (?<upstream_fail>\S+) (?<downstream_local>\S+) (?<downstream_remote>\S+) (?<tls_session_id>\S+) (?<tls_cipher>\S+) (?<tls_sni>\S+) (?<local_cert>\S+) (?<peer_cert>\S+) (?<peer_subject>\S+) (?<peer_issuer>\S+) (?<peer_serial>\S+) (?<not_before>\S+) (?<not_after>\S+) (?<tls_error>\S+)$
    Time_Key    start_time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

[OUTPUT]
    Name            prometheus_exporter
    Match           envoy.access
    Listen          0.0.0.0
    Port            2020
    Metrics         mtls_peer_cert_time_to_expiry_seconds
    # Add additional metric definitions as needed

The parser extracts the not_after field; a downstream Fluent Bit filter (or a Prometheus exporter script) can compute the time‑to‑expiry gauge and expose it for alerting.

By combining these access‑log fields, counter families, and certificate‑freshness signals within a tight correlation window (e.g., 5‑second buckets), operators can quickly determine whether an outage originates in Envoy or the mTLS path, independent of application logs or traces that may be missing during a failure.