Conntrack expiration replays for action-happy models

Introduction to Packet-Walk Benchmark

Overview of NAT and Conntrack

Network Address Translation (NAT) rewrites IP addresses (and optionally ports) as packets traverse a Linux host. The connection tracking subsystem (conntrack) maintains per‑flow state so that NAT can map return traffic correctly. A conntrack entry is created on the first packet of a flow (usually a TCP SYN) and is destroyed when the flow ends or when its lifetime expires. The lifetime is governed by timers that depend on the protocol and the current state (e.g., TCP ESTABLISHED vs. TIME_WAIT).

When NAT and conntrack are used together, a packet’s journey looks like this (simplified):

1. Ingress – NIC → driver → XDP (if present) → ingress qdisc → prerouting (nat table) → conntrack lookup/creation → forward (routing) → postrouting (nat table) → egress qdisc → NIC → wire.
2. Egress (return) – reverse path repeats the same hooks, but the packet now matches an existing conntrack entry, so NAT uses the stored mapping instead of creating a new one.

If any step misbehaves—e.g., the return packet takes a different route, the conntrack entry times out before the flow finishes, or retransmits are mishandled—you observe timeout failures, spurious retransmits, or connection resets.

Importance of Benchmarking

Benchmarking the packet‑walk for NAT/conntrack reveals where latency, CPU overhead, or state‑table pressure originates. By injecting a controlled flow and measuring:

• Per‑packet processing time (XDP → NIC → kernel → NIC)
• Conntrack entry lifetime (how long the entry stays alive)
• Asymmetric return‑path detection (whether the reply sees the same conntrack tuple)
• Retry/retransmit handling (how the kernel reacts to lost SYN/SYN‑ACK)

you can pinpoint whether failures are due to mis‑configured timers, hash‑table contention, routing asymmetry, or excessive lock contention. The benchmark thus serves as a reproducible basis for tuning knobs, validating patches, or comparing offload strategies.

Understanding State Lifetimes

Definition and Purpose of State Lifetimes

A conntrack state lifetime is the interval after which the kernel garbage‑collects a tracking entry if no traffic matching the entry is observed. The purpose is to reclaim memory and prevent stale NAT mappings that could cause mis‑delivery or security issues. Each protocol/state has a default timeout (e.g., TCP ESTABLISHED = 432000 s, UDP = 30 s) defined in /proc/sys/net/netfilter/nf_conntrack_*_timeout.

Factors Affecting State Lifetimes

Measuring State Lifetimes in NAT and Conntrack

1. Direct inspection – Watch /proc/net/nf_conntrack (or conntrack -L) for entry appearance and disappearance timestamps.
2. Kernel tracepoints – Use tracepoint:nf_conntrack_event or kprobe:nf_ct_delete to log lifetime.
3. eBPF histogram – Attach to nf_ct_get and nf_ct_put to measure time between get and put.
4. Packet‑level measurement – Send a packet, then pause traffic for a known interval, and observe whether a reply still gets NAT‑translated.

Example CLI measurement (Linux 5.15+):

# 1. Reset conntrack table (requires root)
sudo conntrack -F

# 2. Start a background listener that will hold a TCP connection open
sudo nc -l -p 9000 &
listener_pid=$!

# 3. From another terminal, initiate a connection and immediately silence traffic
timeout 2 bash -c "exec 3<> /dev/tcp/127.0.0.1/9000; echo -e 'GET / HTTP/1.0\r\n\r\n' >&3; cat <&3" &
# 4. Watch the conntrack entry lifetime
watch -n 1 "sudo conntrack -L | grep ':9000' | awk '{print \$5,\$6,\$7,\$8}'"

The output shows the timer counting down; when it reaches zero the entry vanishes. Adjust net.netfilter.nf_conntrack_tcp_timeout_established and repeat to see the effect.

Asymmetric Return Paths

Definition and Impact of Asymmetric Return Paths

An asymmetric return path occurs when the packet flow from client → server and the reply flow server → client traverse different routes (different next hops, interfaces, or NAT devices). Linux conntrack expects both directions to see the same 5‑tuple (src IP, dst IP, src port, dst port, protocol). If the reply arrives on a different interface or with a different source IP due to routing, the kernel may:

• Fail to find a matching conntrack entry → treat the packet as INVALID and drop it.
• Create a new entry with the reversed tuple, causing the original NAT mapping to be lost and subsequent packets to be mis‑translated.
• Generate spurious state‑table churn, increasing CPU load and memory pressure.

Identifying Asymmetric Return Paths in Packet Flow

To detect asymmetry you must compare the ingress and egress interfaces (or marks) seen by the conntrack entry for each direction.

Methods:

Symptom: A TCP connection that stalls after SYN‑ACK; tcpdump shows SYN‑ACK leaving the server but never arriving at the client, while conntrack -L shows the entry in SYN_RECV state with zero packets in the reply direction.

Mitigating Asymmetric Return Path Issues

1. Enable loose mode – net.netfilter.nf_conntrack_loose = 1 allows conntrack to match packets based on only the source/dest IP and ports, ignoring the interface. This prevents drops when routes differ, but can weaken security (spoofing).
2. Symmetric routing – Ensure return traffic uses the same path (e.g., via policy‑based routing, ECMP hashing pinned to source IP, or using a single uplink).
3. Connection‑tracking zones – Assign flows to a zone (ct zone <id>) and set nf_conntrack_zones to isolate asymmetric traffic.
4. Bypass conntrack for known symmetric flows – Use XDP or nf_flow_table to offload established TCP/UDP flows, eliminating the need for conntrack lookup on the reply path.
5. Adjust nf_conntrack_tcp_behavior – Setting to 2 (Liberal) makes the kernel more tolerant of out‑of‑window packets, reducing false INVALID drops.

Retry Behavior and Timeout Failures

Understanding Retry Behavior in NAT and Conntrack

Linux TCP stack retransmits SYN, SYN‑ACK, and data segments based on tcp_retries* and tcp_syn_retries. Conntrack does not perform its own retransmits; it merely tracks the state. However, timeout failures arise when:

• The SYN (or SYN‑ACK) is lost, the sender retransmits, but the original conntrack entry has already expired (due to a short tcp_timeout_syn_recv or tcp_timeout_syn_sent).
• Asymmetric routing causes the reply to miss the conntrack entry, leading the sender to retransmit SYN‑ACK repeatedly until its retry limit is hit.
• NAT port‑exhaustion forces the kernel to allocate a new source port on retransmit, creating a mismatch with the original conntrack expectation.

Analyzing Timeout Failures and Their Causes

1. Check SYN retransmit count – netstat -s | grep retrans or ss -s shows TCPRetransFail.
2. Inspect conntrack state at failure moment – Use a tracepoint to log nf_ct_delete when state is SYN_SENT or SYN_RECV.
3. Correlate with NAT port allocation – cat /proc/sys/net/ipv4/ip_conntrack_tcp_timeout_max_retrans (if available) or look at nf_nat_redirect logs.
4. Measure round‑trip time (RTT) vs. timeout – If RTT > configured timeout, retransmits will be seen as failures.

Example bpftrace script to catch premature conntrack deletions:

sudo bpftrace -e '
tracepoint:nf_conntrack_event_delete
{
    printf("%s %s %s -> %s [%s] deleted after %d ms\n",
           nfproto_str(args->nfgen_family),
           args->tuple.src.ipv4, args->tuple.src.u3.all,
           args->tuple.dst.ipv4, args->tuple.dst.u3.all,
           ntohl(args->timeout));
}
'

If you see deletions with timeout values far below the expected tcp_timeout_syn_recv (default 60 s) under load, the timer is being aggressively trimmed—often a symptom of hash‑table pressure causing early garbage collection.

Configuring Retry Behavior for Optimal Performance

After adjusting, verify with the packet‑walk benchmark (see later) that retransmit counts drop and connection success rate rises to >99.9 %.

Troubleshooting NAT and Conntrack Timeout Failures

Identifying Symptoms of Timeout Failures

• Application‑level: Clients report “Connection timed out” or “Connection reset by peer” after a few seconds of idle time.
• TCP layer: ss -s shows rising TCPRetransFail and TCPTimeouts.
• Conntrack layer: conntrack -L shows many entries in SYN_SENT or SYN_RECV state with low packet counts, or a high rate of DELETE events.
• NAT layer: iptables -t nat -L -v -n shows increasing DNAT/SNAT packet counters but low RETURN or ACCEPT on reply direction.
• System logs: kernel: nf_ct_ftp: dropping packet, expectation not found or similar helper errors.

Using CLI Tools for Troubleshooting