Canarying MTU fixes without creating new loss

Introduction to MTU Remediation

Understanding MTU and Its Impact

Maximum Transmission Unit (MTU) defines the largest IP packet size that can be transmitted without fragmentation on a given link. When the MTU is mismatched across a path, IP packets that exceed the smallest MTU are either fragmented (if the Don’t‑Fragment (DF) bit is clear) or dropped (if DF is set). In modern data‑center and service‑provider networks, applications often rely on Path MTU Discovery (PMTUD) to avoid fragmentation, but PMTUD can fail when ICMP “Fragmentation Needed” messages are blocked or rate‑limited. The result is silent packet loss, increased latency, and application timeouts—symptoms that are frequently mistaken for congestion or hardware faults.

Importance of Staged Rollout

Changing MTU values is a network‑wide configuration change that touches the data plane of every device in the affected path. A mis‑configured MTU can cause:

• Blackhole traffic – packets larger than the new MTU are dropped silently.
• Increased retransmits – TCP stacks retransmit segments after timeout, inflating load.
• Application‑level failures – protocols that assume a certain payload size (e.g., NFS, iSCSI, VPN tunnels) may break.

Because the failure mode is often invisible until traffic exceeds the threshold, an all‑at‑once push risks a large blast radius with little early warning. A staged rollout introduces explicit transaction boundaries, pre‑checks, verification gates, and operator intervention points that allow the change to be halted before it propagates to the entire fabric.

Pre-Rollout Checks and Planning

Network Assessment and MTU Detection

1. Baseline MTU inventory – Gather the current MTU setting on every L2/L3 interface that participates in the target traffic flow.
   CLI examples:

   # Cisco IOS/XE
   show interfaces | include MTU
   
   # Juniper Junos
   show interfaces terse | match mtu
   
   # Arista EOS
   show interfaces | include MTU
   
   # Linux host
   ip link show | awk '/mtu/ {print $2, $5}'

   Export to a CSV for later comparison.

2. Path MTU discovery – From representative sources (e.g., monitoring hosts, load balancers) run probing tools toward representative destinations across each fabric slice.

   # Linux tracepath with DF bit
   tracepath -n 10.0.0.5
   
   # Cisco IOS ping with DF and sweep
   ping 10.0.0.5 repeat 100 size 1500 df-bit timeout 2

   Record the smallest MTU that yields zero loss. This becomes the candidate safe MTU for the segment.

3. Identify MTU‑sensitive applications – Consult application owners for protocols that embed fixed payload sizes (e.g., Jumbo Frame‑required storage, VXLAN encapsulation, GRE/IPsec tunnels). Note any required overhead (e.g., VXLAN adds 50 bytes).

Identifying Potential Problem Areas

• ICMP filtering – Verify that devices along the path allow ICMP Type 3 Code 4 (Fragmentation Needed) inbound. Use ACL checks or show access-lists to confirm.
• Asymmetric paths – In load‑balanced or multi‑homed environments, the forward and reverse paths may have different MTUs. Perform bidirectional probing.
• Encapsulation overhead – If the transport involves tunneling (e.g., MPLS, VXLAN, GRE), subtract the encapsulation size from the physical MTU to derive the payload MTU that applications see.
• Legacy devices – Note any hardware that cannot support MTU > 1500 (e.g., older NICs, certain ISP CPE). These become natural blast‑radius limits.

Establishing Rollback Thresholds and Criteria

Define measurable, observable conditions that trigger an automatic or operator‑guided rollback. Thresholds must be operational (based on counters you can poll) rather than purely theoretical.

If any metric crosses its threshold within the verification gate after a change, the rollout must halt and initiate rollback for the affected batch.

Implementing Packet-Size Probes

Understanding Packet-Size Probes

A packet‑size probe is a synthetic traffic stream that varies the IP payload length while setting the DF bit to discover the largest packet that traverses the path without loss. Probes can be:

• ICMP Echo (ping -M do -s <size>) – simple, widely supported.
• TCP SYN with MSS negotiation – useful when ICMP is blocked.
• UDP with application‑specific payload (e.g., NFS read) – validates real‑world traffic.

The probe must be rate‑limited to avoid adding load; a typical rate is 1 probe per second per source‑destination pair.

Configuring Packet-Size Probes

Cisco IOS/XE (using IP SLA)

! Define an ICMP echo operation with DF bit and variable size
ip sla 10
 icmp-echo 10.0.0.5 source-ip 10.0.0.1
  frequency 10
  threshold 5000
  timeout 2000
  request-data-size 1400   ! start size
  verify-data
  df-bit enable
!
ip sla schedule 10 life forever start-time now

To sweep sizes, embed the operation in a TCL or EEM script that increments request-data-size and records success/failure.

Juniper Junos (using RPM probe)

set services rpm probe MTU-PROBE target 10.0.0.5 source-address 10.0.0.1
set services rpm probe MTU-PROBE test-type icmp-ping
set services rpm probe MTU-PROBE probe-count 1
set services rpm probe MTU-PROBE test-interval 10
set services rpm probe MTU-PROBE data-size 1400
set services rpm probe MTU-PROBE df-bit enable
set services rpm probe MTU-PROBE thresholds successive-loss 2

A Python script can iterate data-size from 1300 to 9000 in steps of 100 and log the highest size with zero loss.

Linux host (bash one‑liner)

for size in $(seq 1300 100 9000); do
  if ping -M do -c 3 -s $size 10.0.0.5 &>/dev/null; then
    echo "OK $size"
  else
    echo "FAIL $size"
    break
  fi
done

The last “OK” size minus 28 (IP+ICMP header) is the payload MTU.

Interpreting Probe Results

• Success – No loss, RTT within expected baseline. Record the size as candidate MTU.
• Failure – Loss or timeout. Decrease size in next iteration.
• Intermittent loss – May indicate rate‑limited ICMP or transient congestion; repeat probe 3× and require ≥ 2 successes to accept.
• Asymmetry – Run probes in both directions; if results differ, note the lower value as the path MTU and investigate the higher‑loss direction.

The probe outcome feeds directly into the pre‑check: if the candidate MTU differs from the device’s current MTU by more than the allowed overhead (e.g., > 50 bytes for tunneling), the change is flagged for review.

Configuring Retransmit Gates

Understanding Retransmit Gates

A retransmit gate is a verification gate that monitors transport‑layer retransmission metrics after an MTU change. If the change induces packet loss, TCP (or other reliable protocols) will increase retransmits; the gate provides an early, application‑agnostic signal before loss becomes visible in loss‑based probes.

Key counters:

• TCP retransmitted segments (tcpRetransSegs via SNMP, netstat -s on Linux)
• Outgoing retransmits per interface (some vendors export ifOutDiscards that rise with retransmit‑induced drops)
• Application‑level retry counts (e.g., NFS retrans RPC stat)

The gate is considered passed if the delta from baseline stays below a pre‑defined percentage for a defined observation window.

Setting Up Retransmit Gates

Linux host (using ss and snmpd)

# Baseline collection
baseline=$(ss -s | awk '/TCP:/ {print $2}')

# After change, poll every 30 sec
while true; do
  current=$(ss -s | awk '/TCP:/ {print $2}')
  delta=$((current - baseline))
  pct=$((delta * 100 / baseline))
  if (( pct > 20 )); then
    logger -t MTU-GATE "Retransmit spike: $pct% > threshold"
    break
  fi
  sleep 30
done

Integrate this loop into an orchestration tool (Ansible, Salt) that runs on each target device before promoting the change to the next batch.

Cisco IOS/XE (using EEM and SNMP)

! Define SNMP community for retransmits
snmp-server community public RO

! EEM policy to watch tcpRetransSegs (OID 1.3.6.1.2.1.6.12)
event manager applet RETRANSMIT-GATE
 event snmp oid 1.3.6.1.2.1.6.12 get-type exact entry-op gt entry-val 100 poll-interval 60
 action 1.0 syslog priority warnings msg "Retransmit gate exceeded threshold"
 action 2.0 cli command "enable"
 action 3.0 cli command "configure terminal"
 action 4.0 cli command "interface GigabitEthernet0/0/0"
 action 5.0 cli command "mtu 1500"   ! example rollback command

The policy triggers when the counter increments by more than 100 since the last poll (adjust baseline per device).

Juniper Junos (using Python script via op script)

#!/usr/bin/env python3
import jnpr.junos
from jnpr.junos.utils.sw import SW
import time

def get_retransmits(dev):
    rsp = dev.rpc.get_interface_information(normalize=True)
    # parse <input-errors> and <output-errors> as proxy
    # For precise TCP retransmits, query SNMP via net-snmp
    # Placeholder:
    return int(rsp.xpath('//input-errors')[0].text)

with jnpr.junos.Device(host='10.0.0.1') as dev:
    dev.open()
    baseline = get_retransmits(dev)
    while True:
        time.sleep(30)
        current = get_retransmits(dev)
        if (current - baseline) * 100 / baseline > 20:
            print("Retransmit gate triggered")
            # invoke rollback via Junos PyEZ config load
            dev.cu.load(path='/var/tmp/rollback_mtu.conf', format='set')
            dev.cu.commit()
            break

Tuning Retransmit Gate Thresholds

• Baseline window – Collect metrics for at least 5 minutes of stable traffic before the change.
• Observation window – After applying MTU, monitor for 2–3 minutes; if the metric stays within threshold, consider the gate passed.
• Percentage vs. absolute – Use percentage for devices with variable load; absolute counters for low‑volume links where % can be noisy.
• Hysteresis – Require the threshold to be breached for two consecutive samples before triggering rollback to avoid flapping on transient spikes.

Staged Rollout Strategy

Phased Implementation Approach

1. Canary batch – Select a small, representative set of devices (e.g., 2 top‑of‑rack switches, 1 edge router, 2 hosts) that cover:
   • Different hardware vendors.
   • Both upstream and downstream paths.
   • Encapsulated and native traffic.
2. Pre‑check execution – Run packet‑size probes and capture baseline retransmit counters on the canary batch.
3. Change boundary – Apply the new MTU value only to the canary batch via a controlled configuration push (e.g., using Ansible mtu module with check_mode first).
4. Verification gate – After change:
   • Run packet‑size probes again; confirm no loss and that the observed MTU matches the target.
   • Observe retransmit gate for the observation window.
   • Check application‑level health (synthetic transactions, logs).
5. Operator intervention point – If any gate fails, alert the on‑call engineer; they must manually approve continuation or initiate rollback.
6. Batch expansion – If the canary passes, increase the batch size (e.g., 10 % of devices, then continue scaling) while repeating the pre‑check/change/verification cycle until the entire fabric is updated.