Debunking the Habit of Validating TC Profiles with Average RTT Alone

By Tariq Hassan, Observability & Telemetry Lead

Myth: “Low average RTT means your traffic shaping is working.”

Many teams repeat the rule‑of‑thumb: if the average round‑trip time (RTT) stays below a target (e.g., 20 ms), the tc qdisc is correctly limiting latency.
This advice works for steady, low‑variance loads but becomes an anti‑pattern under bursty traffic, micro‑bursts, or when the shaper is mis‑configured for the actual queue discipline. Average RTT can stay low while a small fraction of packets experience large queuing delays, drops, or re‑ordering—exactly the conditions that degrade user‑perceived performance.

1. Introduction to Traffic Control (TC) Profiles

Linux TC lets you attach a qdisc (queueing discipline) to an interface or class. Common choices include htb, fq_codel, codel, tbf, netem, and cake. Each qdisc exposes statistics that describe how packets are queued, delayed, dropped, or re‑marked.

A typical operational question:

Is the tc profile enforcing the intended latency and loss characteristics under real‑world load?

To answer it we look at three orthogonal signals:

1. Latency distribution (not just the average).
2. Queue occupancy (how many packets/bytes are waiting).
3. Burst timing (how quickly the queue builds and drains).

2. Limitations of Average RTT

2.1 Data‑first view

# Average RTT over the last 5 minutes (from an application‑side probe)
avg_over_time(rtt_seconds[5m])

A dashboard may show a flat line at ~8 ms, suggesting everything is fine.

2.2 Why the average hides problems

• Tail latency – A small percentage of packets can wait far longer while the mean stays low.
• Queue absorption – A shallow queue can absorb bursts without raising the average RTT noticeably, yet the queue length may be near its limit.
• Loss vs. delay trade‑off – Some qdiscs (e.g., tbf) prefer dropping excess packets rather than queuing them; average RTT stays low while loss spikes.

2.3 CLI evidence

# Show statistics for the root qdisc on eth0
tc -s qdisc show dev eth0

Sample output (truncated):

qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 0 direct_bytes_stat 0
 Sent 12345678 bytes 9876 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

Notice the backlog field (bytes) and packets (p). If backlog stays at zero while the application reports occasional high latency, the problem lies elsewhere (e.g., in a child class or in the NIC’s internal rings).

2.4 Missing signals

Standard tc exporters often export only tc_queue_len_packets and tc_queue_len_bytes. They rarely expose:

• tc_overlimits – number of times the qdisc exceeded its rate limit.
• tc_drops – packets dropped due to queue limits.
• tc_backlog_bytes – instantaneous backlog (different from the static limit).

Without these, you cannot tell whether latency spikes are caused by queuing or by deliberate loss.

3. Role of Queue Occupancy

3.1 Data‑first view

# Current queue length in packets (from a tc exporter)
tc_queue_len_packets{device="eth0", qdisc="htb"}

A time‑series may linger at 10–20 packets most of the time, then jump to 200 packets for a few seconds while average RTT stays unchanged.

3.2 Mapping to mechanism

• Queue length directly reflects how many packets are waiting to be transmitted.
• For work‑conserving qdiscs (htb, fq_codel), queuing delay ≈ queue_length * packet_size / link_rate.
• A rising queue length predicts forthcoming latency increase before the average RTT reacts.

3.3 CLI example – monitoring burst buildup

# Sample the qdisc every 0.5 s and display backlog
while true; do
  tc -s qdisc show dev eth0 | grep -E 'backlog|Sent'
  sleep 0.5
done

You’ll see the backlog field climb in steps that correspond to application bursts, even when the Sent byte counter changes smoothly.

3.4 Correlating with logs/traces

• Application logs – Look for timestamps where latency spikes are logged (e.g., “request took 120 ms”).
• Distributed traces – Span attributes like queueing_time (if instrumented) will mirror the tc backlog curve.
• NetFlow / sFlow – Sudden increases in packets per flow match the tc backlog rise.

When all three sources show a concurrent rise, you have high confidence that the tc queue is the culprit.

4. Burst Timing

4.1 Data‑first view

# Rate of packets entering the qdisc (approximation via tc stats)
increase(tc_queue_len_packets[10s])

A sharp positive slope indicates a burst; a negative slope indicates drain.

4.2 Why burst timing matters

• Micro‑bursts (sub‑millisecond spikes) can overflow shallow queues before the averaging window of RTT captures them.
• Burst‑size vs. rate – A shaper configured for a steady rate may still allow bursts larger than the bucket size (burst or cburst in htb).
• Latency jitter – Variance in inter‑departure time causes jitter-sensitive applications (VoIP, gaming) to suffer even if mean latency is low.

4.3 CLI – capturing burst dynamics

# Show the current rate and burst counters for an htb class
tc -s class show dev eth0 classid 1:10

Output includes:

class htb 1:10 prio 0 quantum 1514 rate 100Mbit ceil 100Mbit burst 15Kb cburst 1500b
 Sent 12345678 bytes 9876 pkt (dropped 0, overlimits 0 requeues 0)
 rate 0bit/s drops 0 overlimits 0 requeues 0 lended 0 borrowed 0

The rate field (instantaneous transmitting rate) will spike during a burst, while overlimits increments if the burst exceeds ceil.

4.4 gNMI subscription example

# gNMI subscribe request (JSON)
{
  "subscribe": [
    {
      "path": {
        "elem": [
          { "name": "interfaces" },
          { "name": "eth0" },
          { "name": "tc" },
          { "name": "qdisc" }
        ]
      },
      "mode": "sample",
      "sample_interval": 2000000000   # 2 s in nanoseconds
    }
  ]
}

The telemetry payload will contain fields like queue-length-packets, backlog-bytes, overlimit-count, enabling real‑time detection of burst‑induced queue growth.

5. Troubleshooting Workflow

1. Collect baseline – Pull tc -s qdisc show and export queue length, drops, overlimits via your metrics pipeline.
2. Identify anomaly – Alert on:
   • tc_queue_len_packets > threshold (e.g., 80 % of limit).
   • tc_overlimits increasing > 0 per interval.
   • Application latency p99 > SLO while avg RTT OK.
3. Correlate – Overlay trace queueing_time and log latency spikes on the same timeline.
4. Inspect qdisc parameters – tc -p qdisc show dev eth0 to verify rate, ceil, burst, cburst.
5. Validate with controlled traffic – Use tc netem or pktgen to inject known bursts and observe the metrics.
6. Remediate – Adjust shaper parameters, increase queue limit, or switch to a more suitable qdisc (e.g., fq_codel for latency‑sensitive workloads).

6. Scaling Limitations

• Polling overhead – Running tc -s qdisc show on hundreds of interfaces every second can add measurable CPU cost. Mitigation:
  • Use eBPF‑based tc stats collectors (e.g., tcstat from the bpftrace toolkit).
  • Sample at a lower frequency and rely on event‑driven notifications via netlink (RTNETLINK answers).
• Counter wrap‑around – On high‑speed NICs (>10 Gbps), 64‑bit counters may still wrap within minutes; ensure your exporter handles wrap correctly.
• Hidden queues – Some drivers maintain internal TX rings not visible via tc; monitor if_tx_queue_len from /sys/class/net/<dev>/statistics/tx_queue_len and NIC‑specific registers.

7. Best Practices

Minimum Telemetry Set for Confident TC Validation

1. Queue length – packets and bytes (tc_queue_len_packets, tc_queue_len_bytes).
2. Backlog – instantaneous bytes waiting (tc_backlog_bytes).
3. Drops – packets dropped due to queue limits (tc_drops).
4. Overlimits – times the shaper exceeded its rate (tc_overlimits).
5. Latency histogram – if the qdisc provides one (e.g., fq_codel’s ecn or delay histogram).
6. Interface TX drops – if_tx_out_drops (to catch driver‑ring loss).
7. Application latency – p99/p99.9 from client‑side or service‑mesh telemetry.

With these signals you can answer the operational question definitively: Is the tc profile enforcing the intended latency and loss characteristics under real‑world load?

End of article.