State changes that separate churn from path loss

Introduction to Network Troubleshooting

When a service reports “connection reset” or “spurious retransmits,” top‑line dashboards typically show only a rise in TCP retransmit rate and a drop in established‑socket count. These symptoms can stem from either (a) genuine packet loss in the network path or (b) the application actively tearing down and recreating sockets (e.g., crash‑loop, idle‑timeout, load‑shedding). Both look identical in aggregate metrics, so finer‑grained kernel observability is required.

The kernel tracepoint inet_sock_set_state fires each time the TCP state machine moves a struct sock from one state to another (e.g., ESTABLISHED → FIN_WAIT1, SYN_RECV → ESTABLISHED). By tracing this point we can see who (PID/comm) caused the transition and when it happened, independent of packet‑level loss.

Retransmit traces come from TCP tracepoints such as tcp_retransmit_skb and tcp_retransmit_synack. Queue artifacts are observable via /sys/class/net/<iface>/statistics/ (tx_drops, rx_drops, tx_queue_len, rx_queue_len) or via tc -s qdisc show. Together they let us separate loss‑induced retransmits from application‑driven state churn.

Distinguishing Genuine Path Loss from Application‑Driven Connection Churn

Top‑Line Dashboard Symptom

A spike in TCPRetransmits/sec coupled with a dip in EstablishedConnections.

• Path loss → retransmit rise accompanied by a stable or slightly increasing TCPOutRsts/sec (kernel sending resets due to loss).
• Application churn → retransmit rise paired with a burst of TCPInErrs (application‑generated RST) and a correlating rise in TCPActiveOpens.

Using inet_sock_set_state for Deeper Insight

Enable the tracepoint and capture the comm/pid for each state change.

• Path loss → state changes are mostly passive (e.g., ESTABLISHED → FIN_WAIT1 triggered by retransmission timeout, no new TCP_ACTIVEOPEN).
• Application churn → burst of CLOSE → LISTEN or ESTABLISHED → CLOSE initiated by the same PID that also generates a high rate of TCP_ACTIVEOPEN shortly after.

Combining Retransmit Traces and Queue Artifacts

Correlate tcp_retransmit_skb spikes with inet_sock_set_state events and interface queue counters.

• Application‑induced loss → retransmits occur without a rise in tx_drops/rx_drops and the socket moves to CLOSE before any retransmit timeout.
• Path‑related loss → retransmits line up with increased tx_drops (or rx_drops) and the socket stays in ESTABLISHED until the retransmit timeout fires.

In‑Depth Examination of inet_sock_set_state

Definition

The tracepoint is defined in trace/events/tcp.h:

TRACE_EVENT(inet_sock_set_state,
    TP_PROTO(struct sock *sk, int state),
    TP_ARGS(sk, state),
    TP_STRUCT__entry(
        __field(  u16,    family        )
        __field(  u16,    state         )
        __field(  u32,    saddr         )
        __field(  u32,    daddr         )
        __field(  u16,    sport         )
        __field(  u16,    dport         )
        __field(  pid_t,  pid           )
        __field(  char[16], comm        )
    ),
    TP_fast_assign(
        __entry->family   = sk->sk_family;
        __entry->state    = state;
        __entry->saddr    = sk->sk_rcv_saddr;
        __entry->daddr    = sk->sk_daddr;
        __entry->sport    = inet_sk(sk)->inet_sport;
        __entry->dport    = inet_sk(sk)->inet_dport;
        __entry->pid      = task_tgid_vnr(current);
        memcpy(__entry->comm, current->comm, sizeof(__entry->comm));
    ),
    TP_printk("family=%u state=%d %pI4:%u->%pI4:%u pid=%d comm=%s",
        __entry->family, __entry->state,
        &__entry->saddr, __entry->sport,
        &__entry->daddr, __entry->dport,
        __entry->pid, __entry->comm)
);

Enabling the Tracepoint

# Mount tracefs if not already mounted
mount -t tracefs nodev /sys/kernel/tracing

# Enable the specific tracepoint
echo 1 > /sys/kernel/tracing/events/tcp/inet_sock_set_state/enable

# Start recording (binary format)
cat /sys/kernel/tracing/trace_pipe > inet_sock_set_state.bin &

Reading with bpftrace (convenient)

bpftrace -e '
tracepoint:tcp:inet_sock_set_state {
    printf("state=%d %s:%d->%s:%d pid=%d comm=%s\n",
        args->state,
        ntop(AF_INET, &args->saddr), args->sport,
        ntop(AF_INET, &args->daddr), args->dport,
        args->pid, args->comm);
}'

What to notice: each line shows the new TCP state, the four‑tuple, and the owning process. A pattern of many state=1 (TCP_CLOSE) followed quickly by state=10 (TCP_LISTEN) from the same PID indicates application‑driven churn.

CLI Alternatives – perf

perf record -e tracepoint:tcp:inet_sock_set_state -a -- sleep 30
perf script | head -20

The perf script output yields the same fields as above.

Retransmit Traces and Queue Artifacts Analysis

Retransmit Tracepoints

• tcp_retransmit_skb – fires when a data segment is retransmitted.
• tcp_retransmit_synack – fires for SYN‑ACK retransmits.
• tcp_retransmit_ack – fires for duplicate ACKs that trigger fast retransmit.

Enable them:

echo 1 > /sys/kernel/tracing/events/tcp/tcp_retransmit_skb/enable
echo 1 > /sys/kernel/tracing/events/tcp/tcp_retransmit_synack/enable

Queue Artifacts

# TX drops
cat /sys/class/net/eth0/statistics/tx_drops
# RX drops
cat /sys/class/net/eth0/statistics/rx_drops
# Current queue length (bytes)
cat /sys/class/net/eth0/statistics/tx_queue_len
# Or via tc
tc -s qdisc show dev eth0

What to notice: a rise in tx_drops concurrent with tcp_retransmit_skb indicates output‑queue overflow (often due to congestion or policer). A rise in rx_drops with retransmits suggests input‑side loss (e.g., NIC receive ring overrun).

Example Correlated Analysis (bpftrace one‑liner)

bpftrace -e '
tracepoint:tcp:tcp_retransmit_skb {
    @retrans[comm] = count();
}
tracepoint:tcp:inet_sock_set_state /args->state == TCP_CLOSE/ {
    @close[comm] = count();
}
interval:s:5 {
    print("Retransmits:", @retrans);
    print("Close events:", @close);
    clear(@retrans); clear(@close);
}'

What to notice: if @retrans and @close spike together for the same comm, the retransmits are likely application‑driven (the app closes sockets, causing retransmits of in‑flight data). If @retrans rises while @close stays flat, the loss is path‑related.

Scaling Limitations and Considerations

Performance Impact of inet_sock_set_state

Enabling many tracepoints adds overhead proportional to event frequency. On a busy host handling >100k TCP state changes/sec, the tracepoint adds ~2‑5 µs per event (measured via perf stat -e tracepoint:tcp:inet_sock_set_state). This translates to 0.2‑0.5 % CPU on a 2 GHz core—acceptable for short troubleshooting windows but not for continuous production monitoring at scale.

Retransmit tracepoints fire less often (only on loss) and thus have lower overhead, but in high‑loss scenarios they can still reach tens of thousands per second.

Queue artifact polling via sysfs is O(1) and negligible; tc -s qdisc show incurs a netlink dump (~10‑30 µs) per interface.

Mitigations

1. Sampling – use perf record -f -c 100000 to sample every Nth event, reducing overhead while preserving statistical accuracy for trend analysis.
2. eBPF maps with per‑CPU counters – aggregate counts in kernel space and expose via /sys/fs/bpf/ to avoid user‑space copy per event.
3. Selective enabling – enable inet_sock_set_state only on a subset of CPUs (/sys/kernel/tracing/tracing_cpumask) or on specific PIDs via pidfilter (available in newer kernels).
4. Time‑bounded collection – limit tracing to the incident window (e.g., start tracing upon alert, stop after 30 s) using a wrapper script that toggles the tracepoint via tracefs.

Troubleshooting Methodologies

Identifying Genuine Path Loss

1. Verify a rise in tcp_retransmit_skb and an increase in either tx_drops or rx_drops on the affected NIC.
2. Confirm that inet_sock_set_state shows mostly passive transitions (ESTABLISHED → FIN_WAIT1, FIN_WAIT1 → CLOSING) without a matching surge in TCP_ACTIVEOPEN from the same PID.
3. Check that retransmit timeout (RTO) values (via ss -ti) are increasing, indicating the kernel is backing off due to perceived loss.
4. Correlate with link‑layer metrics (e.g., ethtool -S eth0 | grep tx_errors) to rule out NIC faults.

Detecting Application‑Driven Connection Churn

1. Observe a burst of inet_sock_set_state events where state transitions CLOSE → LISTEN or ESTABLISHED → CLOSE originating from the same PID/comm.
2. See a concurrent rise in tcp_retransmit_skb but no increase in NIC drop counters.
3. Note a spike in TCP_ACTIVEOPEN (from netstat -s or ss -s) matching the PID’s new socket creations.
4. Look at application logs for explicit close()/shutdown() loops or idle‑timeout logic.

Step‑by‑Step Guide

Following this workflow lets you distinguish whether a spike in retransmits and connection churn originates from network path loss or from application‑driven socket churn, enabling targeted remediation.