Disentangling Neighbor Discovery Warm‑up, ARP Suppression, Conntrack Expectations, and Delayed Interface Readiness

Neighbor Discovery Protocol Overview

Neighbor Discovery Protocol (NDP) replaces ARP in IPv6 and provides address resolution, neighbor unreachability detection, duplicate address detection, and router discovery. NDP messages are carried in ICMPv6 packets: Neighbor Solicitation (NS), Neighbor Advertisement (NA), Router Solicitation (RS), Router Advertisement (RA), and Redirect. A host learns the link‑layer address of a neighbor by sending an NS to the solicited‑node multicast address and waiting for an NA. The process includes a warm‑up phase where the host may retransmit NS up to MAX_MULTICAST_SOLICIT (default 3) times with exponential back‑off before giving up.

ARP Suppression Mechanisms and Benefits

ARP suppression prevents unnecessary ARP requests in fabrics where the IP‑to‑MAC mapping is known a priori (e.g., overlay networks, data‑center leaf‑spine designs). By suppressing ARP, the control plane reduces broadcast traffic, lowers ARP table churn, and speeds up host‑to‑host communication.

Common implementations:

• Proxy ARP – a device answers ARP requests on behalf of another host.
• Linux bridge ARP suppression – bridge link set dev <vxlan> arp_suppress on.
• EBTables/NFTables rules – drop ARP requests for known IP/MAC pairs.
• Vendor‑specific ASIC features – Cisco NX‑OS arp suppress, Juniper arp-suppression.

When ARP suppression is active, the host will not see an ARP request for a suppressed address; instead, it relies on pre‑populated neighbor entries or proxy replies.

Conntrack Fundamentals and Expectations

Linux connection tracking (conntrack) maintains state for packets traversing the firewall/NAT subsystem. Each tracked connection is represented by a tuple (src IP, dst IP, src port, dst port, protocol) and a state (NEW, ESTABLISHED, RELATED, INVALID, etc.). Conntrack can also hold expectations: pre‑created entries that match packets before they arrive, allowing the kernel to accept packets that would otherwise be dropped as INVALID. Expectations are useful for protocols that open related connections (FTP, SIP) or for test traffic that must bypass the normal NEW→ESTABLISHED transition.

Key sysctls:

• net.netfilter.nf_conntrack_max – maximum tracked entries.
• net.netfilter.nf_conntrack_tcp_timeout_established – idle timeout for ESTABLISHED TCP.
• net.netfilter.nf_conntrack_expect_max – maximum number of expectations.

Interface Readiness and Delayed Initialization

An interface is considered ready when:

1. The device driver reports carrier (physical link) or operstate = UP.
2. The kernel has assigned an IP address and completed any required link‑layer setup (e.g., VLAN tagging, MAC learning).
3. For virtual interfaces (veth, macvlan, SR‑IOV VFs), the peer namespace or device must also be up.

Delay can arise from:

• Asynchronous bring‑up – e.g., DHCP client still negotiating when the first probe is sent.
• Hardware latency – SFP+ modules needing several milliseconds to assert LOS/LOS.
• Software‑defined interfaces – VXLAN devices wait for the underlying UDP socket to be bound before declaring readiness.
• Kernel race conditions – when a device is added via ip link add and immediately probed before the driver finishes registering net_device callbacks.

Symptom: the first ICMP echo or TCP SYN probe fails with “Network is unreachable” or “No route to host”, while subsequent probes succeed after a short pause (typically 100‑500 ms).

Troubleshooting Pre‑Deployment Probe Failures

Identifying the Root Cause

1. Capture the failure – run a continuous ping with timestamps:

   ping -i 0.2 -D 2001:db8::1 | ts

   Look for the first line showing Destination Host Unreachable or 100% packet loss.
2. Check neighbor state – immediately after the failed probe, inspect the neighbor cache:

   ip -6 neigh show dev eth0
   arp -n -i eth0

   If the entry is INCOMPLETE or absent, the failure is likely at the L2/L3 resolution stage.
3. Verify conntrack expectations – list active expectations:

   conntrack -L -p tcp --dport 80 | grep EXPECT

   A missing expectation can cause the first SYN to be dropped as INVALID.
4. Confirm interface readiness – query carrier and operstate:

   ip link show dev eth0
   ethtool eth0 | grep -E "Link detected|Speed"

   If carrier is no or operstate is LOWERLAYERDOWN, the interface is not yet ready.

Analyzing NDP and ARP Suppression Logs

• NDP debugging – enable kernel NDP logging via sysctl:

  sysctl -w net.ipv6.conf.all.debug=1
  dmesg -w | grep -i "ndp\|neighbor"

  Look for repeated NS transmissions without a matching NA.
• ARP suppression logs – on a Linux bridge, enable bridge netdev notifications:

  echo 1 > /sys/class/net/br0/bridge/log_level
  journalctl -u systemd-udev -f | grep -i "arp_suppress"

  Suppressed ARP requests appear as “ARP request dropped (suppressed)”.

If the first probe sees repeated NS/ARP requests with no reply, the issue is likely that the neighbor entry has not yet been created because suppression is preventing the request from reaching the target, or the target is not yet ready to answer.

Conntrack and Interface Readiness Troubleshooting Techniques

• Expectation verification – simulate the expected traffic with nc or hping3 and watch conntrack:

  conntrack -E -p tcp --dport 80 &
  hping3 -S -p 80 -c 1 2001:db8::1

  If the expectation is hit, you’ll see a NEW entry transition directly to ESTABLISHED.
• Interface readiness polling – script a loop that waits for carrier:

  while ! ip link show dev eth0 | grep -q "state UP"; do
      sleep 0.05
  done
  echo "Interface ready"

• Conntrack table exhaustion – check for drops:

  cat /proc/net/stat/nf_conntrack

  Look for entries approaching max and dropped non‑zero.

Disentangling NDP Warm‑up and ARP Suppression

NDP Warm‑up Process and Timers

NDP warm‑up follows RFC 4862 §5.3:

1. Initial delay – a random delay between 0 and MAX_RTR_SOLICIT_DELAY (default 1 s) before sending the first RS.
2. Retransmission timer – RetransTimer defaults to 0 (retransmits only for duplicate address detection). For NS/NA, the retransmission interval is RetransTimer (if set) or the default 0 ms (no retransmit); however, the protocol specifies a maximum of MAX_MULTICAST_SOLICIT = 3 NS transmissions with exponential back‑off (initial interval 0 s, then 1 s, then 2 s).
3. Duplicate Address Detection (DAD) – sends up to MAX_DUP_ADDR_DETECT (default 1) NS with the tentative address; if no NA is received, the address is considered unique.

If the first probe is sent before DAD completes, the source address may be tentative, causing the peer to drop the packet as it cannot verify uniqueness.

ARP Suppression and its Impact on Neighbor Discovery

ARP suppression does not affect IPv6 NDP directly, but in dual‑stack environments the suppression of ARP can cause a host to rely on IPv6 for reachability. If NDP warm‑up is delayed (e.g., due to suppressed RS/RA or missing proxy NDP), the IPv6 path may not be ready, leaving only the suppressed ARP path—which is intentionally silent—resulting in a black hole for the first probe.

• Proxy NDP – enabled via sysctl -w net.ipv6.conf.all.proxy_ndp=1. When active, the kernel answers NS for addresses it does not own but has a route to, mimicking proxy ARP.
• NDP suppression – some platforms (e.g., Cisco Nexus) allow ipv6 nd suppress on an interface to stop sending RA/RS; this can delay host autoconfiguration.

Thus, the first probe failure can be traced to either:

• ARP suppression preventing the ARP request from reaching the target (target never sees the request).
• NDP warm‑up delay causing the source to use a tentative address or missing router advertisement.

Code Examples for Configuring ARP Suppression and Neighbor Discovery

Enable ARP suppression on a VXLAN bridge (Linux kernel ≥ 4.2):

# Create VXLAN interface
ip link add vxlan10 type vxlan id 10 dev eth0 dstport 4789
# Bring up bridge and enslave VXLAN
ip link add name br0 type bridge
ip link set dev vxlan10 master br0
# Enable ARP suppression on the bridge
ip link set dev br0 type bridge arp_suppress on
# Bring up the bridge
ip link set dev br0 up

Enable proxy NDP on a host:

sysctl -w net.ipv6.conf.all.proxy_ndp=1

Disable NDP suppression on a Cisco Nexus interface (example):

interface Ethernet1/1
  no ipv6 nd suppress