When NIC offloads make counters disagree

Introduction

Packet capture, interface counters, and Prometheus series are core tools for network monitoring, yet they can diverge when Generic Receive Offloading (GRO), TCP Segmentation Offloading (TSO), or hardware offloads are active. This article explains why each signal may misrepresent traffic under those conditions and which source is “lying” in each case.

GRO (Generic Receive Offloading)

What it does: GRO merges multiple incoming packets into a single larger packet before they reach the network stack, reducing per‑packet CPU overhead.

Effect on Packet Capture

Tools like tcpdump or Wireshark see the aggregated packet, not the original frames. Consequently, capture files may miss individual packets, leading to under‑counting of events such as retransmissions or application‑layer messages.

# Enable GRO on eth0
ethtool -K eth0 gro on

Effect on Interface Counters

The kernel counts the merged packet as one RX packet. Interface counters (ip -s link show) therefore under‑report the true packet arrival rate.

Effect on Prometheus Series

Metrics derived from interface counters (e.g., ip_link_rx_packets) inherit the same under‑count, so Prometheus graphs will show lower traffic than actually received.

Signal that is lying: Interface counters and any Prometheus series built from them under‑report; packet capture is also incomplete but shows the combined packet correctly.

TSO (TCP Segmentation Offloading)

What it does: TSO lets the NIC split a large TCP segment into MTU‑sized frames for transmission, moving segmentation work from the CPU to hardware.

Effect on Packet Capture

Capture tools observe the post‑segmentation frames, not the original large TCP segment. If you expect to see a single large packet, you will instead see many smaller ones, which can confuse flow‑reassembly or application‑level analysis.

# Enable TSO on eth0
ethtool -K eth0 tso on

Effect on Interface Counters

Each segmented frame is counted as a separate TX packet, so counters over‑report the number of TCP segments transmitted by the stack.

Effect on Prometheus Series

TX‑related metrics (e.g., ip_link_tx_packets) will appear inflated relative to the application’s view of traffic.

Signal that is lying: Interface counters and Prometheus series over‑report TX packets; packet capture accurately reflects what actually went on the wire (the segmented frames).

Hardware Offloads (checksum, VLAN insertion, etc.)

What they do: Specific packet‑processing tasks are delegated to the NIC, bypassing the host CPU.

Effect on Packet Capture

If the capture point is before the NIC (e.g., using a tap or mirror port), you will see the raw packets as the NIC receives them—checksums may be zero, VLAN tags absent, etc. Captures taken after the NIC (e.g., via tcpdump on the host interface) may show processed packets, leading to inconsistency depending on capture location.

# View current offload settings
ethtool -k eth0

Effect on Interface Counters

Counters still count packets as they enter/leave the NIC, so they remain accurate for byte and packet totals. However, derived metrics that assume CPU‑performed work (e.g., checksum validation) can be misleading.

Effect on Prometheus Series

Since most Prometheus node‑exporter metrics come from the same counters, they generally stay correct for volume but may misrepresent processing load.

Signal that is lying: Packet capture can be misleading if taken at the wrong point; interface counters and Prometheus series remain reliable for volume but not for verifying offload correctness.

Troubleshooting Divergent Signals

Check offload state

ethtool -k eth0 | grep -E 'gro|tso|rx|tx'

Capture at multiple points
- Pre‑NIC (tap/mirror) to see what the NIC receives.
- Post‑NIC (host interface) to see what the stack processes.

Compare counters vs. capture

# RX packets from counters
ip -s link show eth0 | grep RX
# Unique packets in a capture (requires post‑processing)
tshark -r capture.pcap -q -z io,stat,0,"COUNT(frame)frame"

Validate with application metrics
If application‑level byte counts match interface counters but not capture, suspect GRO/TSO or capture‑point issues.
Disable offloads temporarily for validation
```
ethtool -K eth0 gro off tso off
```
Observe whether counters and capture converge.

Best Practices

Always verify capture location relative to where offloads apply.
Use multiple data sources (counters, capture, application logs) to triangulate truth.
Disable specific offloads when debugging packet‑level anomalies.
Scale Prometheus via federation or remote write if cardinality from high‑resolution counters becomes a concern.
Document offload settings in your monitoring baseline so deviations are immediately noticeable.

Conclusion

Under GRO, interface counters and Prometheus series under‑report packets; packet capture shows aggregated frames but misses individual ones.
Under TSO, counters and Prometheus over‑report TX packets; capture accurately reflects the segmented wire traffic.
With hardware offloads, packet capture can be misleading depending on capture point, while counters and Prometheus remain accurate for volume but not for verifying offload correctness.

By checking offload states, capturing at appropriate points, and cross‑validating with application metrics, you can determine which signal is trustworthy and adjust your monitoring strategy accordingly.