Listener warming failures behind healthy endpoints

Introduction to Envoy and Service Discovery

Overview of Envoy Architecture

Envoy is a high‑performance L7 proxy built around a modular filter chain. Its core includes a dispatcher (event loop), a connection manager per listener, and worker threads that execute the filter chain. Configuration is delivered dynamically via xDS APIs: LDS (Listener Discovery Service), RDS (Route Discovery Service), CDS (Cluster Discovery Service), and EDS (Endpoint Discovery Service). The control plane pushes updates over gRPC or REST‑JSON; Envoy maintains a versioned configuration snapshot and applies changes atomically when a new version arrives. Listener warming prepares a newly added or modified listener before it accepts traffic, initializing filters, connection buffers, and thread‑local caches.

Service Discovery Mechanisms in Envoy

Envoy separates traffic routing from endpoint health through layered discovery:

LDS – supplies listener objects (address, filter chain, per‑listener stats).
RDS – delivers virtual host and route configurations bound to a listener.
CDS – provides cluster definitions (load‑balancing policy, circuit breakers, health‑checking).
EDS – offers endpoint IP:port pairs for each cluster, including health status from active or passive checks.

When an xSD stream updates, Envoy increments the version for that resource type, validates the new snapshot, and, if valid, swaps the active configuration. Existing connections continue uninterrupted; new connections use the updated configuration after listener warming completes.

Understanding 503 NR Errors in Envoy

Definition and Causes of 503 NR Errors

Envoy returns HTTP 503 NR (No Route) when a request reaches a listener but no matching route exists in the active RDS configuration. The “NR” suffix distinguishes it from other 503 variants (e.g., 503 UC for upstream connect failures, 503 OFL for overflow). Typical causes:

Listener exists but its associated virtual host/route table is empty or missing.
RDS update has not yet been applied (config propagation lag).
Listener warming is incomplete, so the router filter is not ready.
Route validation fails (e.g., missing cluster reference) and Envoy discards the route during installation.

Role of LDS and RDS in Envoy Configuration

LDS determines where Envoy listens; RDS determines what to do with traffic once a connection is accepted. If LDS updates faster than RDS, a listener may be active while its route table is still stale or empty, leading to 503 NR. Conversely, if RDS updates before the listener is warmed, Envoy may install routes that reference a listener not yet ready to accept connections, producing the same symptom after the listener becomes active.

LDS and RDS Update Order and Listener Warming

Listener Warming Semantics in Envoy

When Envoy receives a new listener version via LDS, it performs the following before marking the listener “warming complete”:

Allocate listener sockets and bind to the address.
Instantiate the filter chain (network filters → HTTP connection manager → router filter).
Initialize thread‑local objects (e.g., rate‑limit counters, stats).
Perform any asynchronous filter initialization (e.g., TLS context loading, JWT provider warm‑up).
Once all filters report readiness, transition the listener state from warming to active and begin accepting new connections.

During warming, existing connections continue using the old listener configuration; new connections are queued or rejected with 503 NR if the router filter cannot resolve a route.

Impact of LDS and RDS Update Order on Service Discovery

Consider this sequence:

Control plane pushes new LDS (listener A) → Envoy starts warming listener A.
Control plane pushes new RDS (routes for listener A) while listener A is still warming.
Envoy applies the RDS update immediately to the pending configuration for listener A.
Listener A finishes warming; the router filter now has the fresh route table and can match incoming requests.

If step 2 occurs after listener A has already become active (i.e., LDS update completed before RDS), the active configuration temporarily lacks routes, causing 503 NR until the RDS update arrives. The reverse order (RDS before LDS) is safe because the router filter holds a reference to a non‑existent listener; Envoy discards the route during validation and reinstalls it once the listener appears.

Example Configuration for LDS and RDS Updates

Below is a minimal static bootstrap that points Envoy to an xDS gRPC server. Dynamic resources are delivered via LDS/RDS.

static_resources:
  listeners: []   # empty – all listeners come via LDS
  clusters:
    - name: xds_cluster
      connect_timeout: 0.25s
      type: LOGICAL_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: xds_cluster
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: xds.example.com
                      port_value: 8080

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

dynamic_resources:
  lds_config:
    resource_api_version: V3
    api_config_source:
      api_type: GRPC
      grpc_services:
        - envoy_grpc:
            cluster_name: xds_cluster
  rds_config:
    resource_api_version: V3
    api_config_source:
      api_type: GRPC
      grpc_services:
        - envoy_grpc:
            cluster_name: xds_cluster

To simulate a lag, pause the xDS server for a few seconds after sending LDS but before sending RDS, then observe 503 NR via the admin interface:

# Query routing stats
curl -s http://localhost:9901/stats | grep listener.http.ingress_http
# Look for upstream_rq_503_nr increments

Troubleshooting 503 NR Errors and Traffic Rejection

Identifying Config Propagation Lag vs Application Failure

Check listener state – GET /listeners admin endpoint shows warming vs active.
Inspect route table – GET /config_dump reveals the active routes array for each listener. Empty or missing virtual host indicates lag.
Correlate timestamps – Compare the last_updated timestamp of the listener resource (from LDS) with that of the route resource (from RDS). A gap > propagation delay suggests lag.
Verify endpoint health – GET /clusters?format=json shows healthy endpoints; if all are healthy, the problem is not upstream.
Application logs – If the service returns 5xx, Envoy will show 503 UC or 503 OFL, not NR.

Using Envoy CLI and Logging for Troubleshooting

Start Envoy with debug logging for the HTTP router and LDS/RDS subsystems:

envoy -c envoy.yaml --log-level debug --log-format '[%Y-%m-%d %T.%e] [%l] %v' \
      --component-log-level http:debug,router:debug,lds:debug,rds:debug

Watch for log lines such as:

[info] [router] [C12345] No route matched for path /foo
[debug] [lds] [C12345] Listener listener_0 updated, version=3
[debug] [rds] [C12345] RouteConfig route_0 updated, version=2
[info] [listener] Listener listener_0 warming complete

Validate static parts of the config before startup:

envoy --mode validate -c envoy.yaml

Code Examples for Debugging LDS and RDS Updates

Python script to fetch and diff LDS/RDS versions via the admin API:

import json, requests, time

ADMIN = "http://localhost:9901"

def get_resources(url):
    r = requests.get(f"{ADMIN}{url}")
    r.raise_for_status()
    return r.json()

while True:
    lds = get_resources("/config_dump?listeners")
    rds = get_resources("/config_dump?routes")
    l_ver = lds["dynamic_state_listeners"][0]["version_info"]
    r_ver = rds["dynamic_state_route_configs"][0]["version_info"]
    print(f"{time.time():.0f} LDS v{l_ver} RDS v{r_ver}")
    if l_ver != r_ver:
        print("*** VERSION MISMATCH – potential 503 NR window ***")
    time.sleep(2)

Envoy filter to inject custom stats for routing misses:

// router_miss_filter.cc
#include "envoy/registry/registry.h"
#include "envoy/server/filter_config.h"

class RouterMissFilter : public Envoy::Http::StreamFilter {
public:
  // ... required callbacks ...
  Envoy::Http::FilterHeadersStatus decodeHeaders(Envoy::Http::RequestHeaderMap&, bool) override {
    if (!route_) {
      stats_.miss_.inc();
      return Envoy::Http::FilterHeadersStatus::StopIteration;
    }
    return Envoy::Http::FilterHeadersStatus::Continue;
  }
  // ...
};
static Envoy::Registry::RegisterFactory<RouterMissFilterConfig,
                                        Envoy::Server::Configuration::NamedHttpFilterConfigFactory>
    register_;

Compile and add to the listener’s filter chain; watch router_miss.miss rise during LDS/RDS lag.

Scaling Limitations and Considerations

Scaling Envoy for Large-Scale Service Discovery

xDS stream multiplexing: Each Envoy instance opens a separate gRPC stream for LDS, RDS, CDS, EDS. At scale (≥10k instances) the control plane must handle hundreds of thousands of concurrent streams; use HTTP/2 connection pooling and enable max_concurrent_streams on the xDS server.
Resource size: Large route tables (hundreds of thousands of entries) cause significant memory overhead and increase the time to serialize/deserialize updates, extending the warming window.
CPU cost of validation: Envoy validates each incoming RDS against the current listener’s cluster references; validation is O(N) in the number of routes.

Limitations of LDS and RDS in High-Traffic Scenarios

Listener warming latency: Complex filter chains (e.g., RBAC, JWT, external auth) can take tens to hundreds of milliseconds to warm. During this window, new connections see 503 NR if routes are missing.
Race condition bursts: Frequent, small updates (e.g., per‑endpoint weight changes) may keep Envoy constantly in a warming state, leading to a persistent low‑level 503 NR error rate.
Admin endpoint overload: Frequent /config_dump queries under high load add latency; rely on stats instead.

Strategies for Mitigating Scaling Limitations in Envoy

Batch updates: Configure the xDS server to send combined LDS+RDS updates in a single transaction (same version number) using the Resource wrapper’s version_info. This eliminates ordering gaps.
Enable listener warming bypass: Set listener.warming_gate to false via the listener’s per_connection_buffer_limit_bytes and drain_type to allow immediate acceptance while warming continues (only safe if filters are idempotent).
Hierarchical routing: Use virtual hosts with inclusive route rules that catch‑all to a fallback cluster, reducing the chance of an empty route table.
Separate control planes: Run a lightweight, high‑frequency CDS/EDS plane for endpoint health and a slower,