Introduction to NIC and Kernel Upgrades
A NIC or kernel upgrade typically follows these steps:
- Pre‑upgrade inventory – record NIC model, firmware, driver (
ethtool -i <iface>), offload flags (ethtool -k <iface>), and kernel release (uname -r). - Package update – install the new kernel and matching NIC driver (via package manager or source compile).
- Reboot – load the new kernel and driver modules.
- Post‑upgrade validation – repeat the inventory, compare to the baseline, and verify datapath behavior.
If any step is omitted, the intended offload configuration (set in /etc/network/interfaces, Netplan, or a CM tool) can diverge from the actual driver state exposed after the upgrade. This divergence appears in capture semantics—checksums, segmentation, and retransmission patterns—often making the same retransmission case look radically different before and after the maintenance window.
Verifying that intended and actual states match is the first line of defense against mysterious retransmission spikes, CPU regressions, or latency issues.
Understanding Intended Offload Settings
Offload Types
| Offload | Direction | What it does | Typical tunables |
|---|---|---|---|
| TSO (TCP Segmentation Offload) | Tx | Kernel gives NIC a large TCP segment (≤64 KB); NIC splits into MSS‑sized frames, adds seq numbers, computes checksum. | net.ipv4.tcp_tso_win_divisor, net.ipv4.tcp_tso_autosize |
| GSO (Generic Segmentation Offload) | Tx | Software fallback for TSO when NIC lacks hardware support; still produces large SKBs for later hardware segmentation. | Same as TSO |
| GRO (Generic Receive Offload) | Rx | Driver coalesces incoming packets into a larger SKB before passing up the stack. | `ethtool -K |
| LRO (Large Receive Offload) | Rx | NIC‑specific; merges consecutive TCP packets into one larger packet before handing to stack. | `ethtool -K |
| TXCSUM / RXCSUM | Tx/Rx | NIC calculates IPv4/TCP/UDP checksums on Tx; on Rx, NIC verifies and optionally strips checksum. | `ethtool -K |
| SCATTER‑GATHER (SG) | Tx/Rx | Allows NIC to use fragmented buffers (multiple pages) for a single packet, reducing copy overhead. | Usually always on; can be disabled via `ethtool -K |
| TXVLAN / RXVLAN | Tx/Rx | NIC inserts/extracts VLAN tags. | `ethtool -K |
| Ntuple filtering | Rx | Directs packets to specific queues based on flow hash. | ethtool -N <iface> flow-type tcp4 … |
| RSS / RPS / XPS | Rx/Tx | Spreads receive queues across CPUs (RSS in NIC, RPS in kernel, XPS for transmit). | /sys/class/net/<iface>/queues/rx-*/rps_cpus, etc. |
Configuring for Optimal Performance
For 10 GbE+ NICs a performance‑oriented baseline typically enables:
- TSO/GSO on Tx (reduces per‑packet CPU cost).
- TXCSUM/RXCSUM (offloads checksum work).
- SG (zero‑copy capability).
- GRO on Rx (if latency tolerance permits; lowers interrupt rate).
- LRO only when the NIC driver explicitly recommends it (many modern drivers favor GRO).
- TXVLAN/RXVLAN when VLAN tagging is used.
- RSS matched to hardware queues and CPU cores.
These settings are persisted via:
ethtool -K <iface> <feature> on|offin/etc/network/interfaces(post‑up stanza) or a Netplanethernets:<iface>:offload:block.- udev rules (e.g.,
ACTION=="add", SUBSYSTEM=="net", KERNEL=="eth*", RUN+="/sbin/ethtool -K $name tso on gso on gro on"). - Configuration management (Ansible
community.general.ethtool, Chefethtool, Puppetethtool).
Tools for Verifying Intended Settings
| Tool | What it shows | Typical usage |
|---|---|---|
ethtool -k <iface> | List of offload features and current on/off state. | Baseline before upgrade. |
ethtool -i <iface> | Driver name, version, firmware version, bus info. | Verify driver/firmware match. |
ethtool -S <iface> | NIC‑specific statistics (tx/tso_bytes, rx/gro_flush, etc.). | Correlate offload usage with counters. |
| `sysctl -a | grep -E ‘tso | gro | lro |
ip link show <iface> | Current MTU, qdisc, and offload flags (<NO-CARRIER,BROADCAST,MULTICAST,UP,LOWER_UP> plus txqueuelen). | Quick sanity check. |
/sys/class/net/<iface>/device/ | Direct access to PCI config, firmware version, driver‑specific attributes. | Low‑level verification. |
A simple validation script:
#!/usr/bin/env bash
IFACE=eth0
BASELINE=$(ethtool -k "$IFACE")
POST=$(ethtool -k "$IFACE")
diff -u <(echo "$BASELINE") <(echo "$POST") && \
echo "Offload state unchanged" || \
echo "Offload drift detected!"If drift appears, re‑apply the intended settings or investigate why the driver/kernel altered them.
Actual Driver State After Upgrade
Verifying Driver Version and Compatibility
After a kernel upgrade the NIC driver may be:
- In‑tree – version tied to the kernel release.
- Out‑of‑tree (DKMS or vendor RPM) – may need recompilation against new kernel headers.
- Firmware‑dependent – certain features (VLAN insertion, checksum offload) require a minimum firmware level.
Check the driver:
ethtool -i eth0
# Example output:
# driver: i40e
# version: 2.9.21
# firmware-version: 6.0 0x80000c21 1.1326.0
# bus-info: 0000:3b:00.0
# supports-statistics: yes
# supports-test: yes
# supports-eeprom-access: yes
# supports-register-dump: yes
# supports-priv-flags: yesIf the driver is out‑of‑tree, verify it loaded cleanly:
dmesg | grep -i eth0 # look for “version magic” mismatches
cat /proc/sys/kernel/modules_disabled # 0 = modules allowed; 1 = disabled (secure boot)Checking Driver Configuration and Settings
Even when the driver loads, its default offload mask may differ from the previous version. Many drivers expose module parameters to set defaults:
systool -v -m i40e | grep -A2 Parameters
# or
modinfo i40e | grep parmIf the driver does not expose offload defaults, the initial state after probe is determined by:
- NIC NVRAM/factory defaults.
- Any
ethtool -Kcommands executed by init scripts or NetworkManager. - Kernel global defaults (
net.core.default_qdisc,net.ipv4.tcp_tso_win_divisor, etc.).
To see what the driver actually programmed into the NIC after probe:
# Capture pre‑load state (if possible via initramfs shell)
ethtool -k eth0 > /tmp/pre.txt
# After boot:
ethtool -k eth0 > /tmp/post.txt
diff -u /tmp/pre.txt /tmp/post.txtA non‑empty diff indicates the driver (or an early‑boot script) altered offloads.
Troubleshooting Driver‑Related Issues After Upgrade
| Symptom | Likely cause | Diagnostic command |
|---|---|---|
| All offloads disabled after upgrade | Driver reset to safe defaults; missing firmware; module param overriding. | `dmesg |
| TX checksum offload missing but RX present | Asymmetric offload support in newer driver; NIC feature bits changed. | ethtool -k eth0 | grep txcsum; compare ethtool -i eth0 bus‑info to vendor datasheet. |
| Increased interrupt rate, CPU spikes | GRO/LRO disabled inadvertently; Rx queue starved. | cat /proc/interrupts | grep eth0; ethtool -S eth0 | grep rx_packets vs rx_gro_flush. |
| Packet loss under burst | TSO disabled causing many small packets; NIC tx ring too small. | ethtool -g eth0 (show ring sizes); increase with ethtool -G eth0 tx 4096. |
| Link flaps after upgrade | Firmware incompatibility; driver not handling new PHY registers. | ethtool eth0; `dmesg |
Address mismatches by re‑applying the intended offload settings (ethtool -K …), updating firmware, or reinstalling the driver against the correct kernel headers.
Bottom line: After any NIC or kernel upgrade, compare the intended offload configuration (what you configured) with the actual driver state (what ethtool -k reports) and validate capture semantics. This practice prevents offload‑drift‑induced retransmission mysteries and ensures predictable performance.