Canarying ARP inspection in mixed host environments

Introduction to ARP Inspection Rollout

Overview of ARP Inspection

ARP Inspection (AI) is a security feature that validates ARP packets against a trusted source—typically DHCP snooping bindings or static ARP ACLs—before forwarding them. When enabled on a switch port or VLAN, the device examines each ARP request/reply, compares the sender MAC and IP addresses to the binding table, and drops packets that fail validation. This prevents ARP spoofing attacks such as man‑in‑the‑middle, DHCP starvation, and IP‑address hijacking. In modern data‑center fabrics, AI is often deployed alongside DHCP snooping, IP source guard, and port‑security to form a layered defense against L2 attacks.

Importance of Safe Rollout Patterns

Enabling ARP Inspection changes the forwarding behavior of every ARP frame that traverses the protected domain. A mis‑configured binding table, an overly aggressive validation mode, or a transient topology change can cause legitimate hosts to lose connectivity, leading to service‑impacting outages. Because ARP is a ubiquitous, low‑latency protocol, any disruption propagates quickly across applications that rely on immediate L2 reachability (e.g., clustered databases, real‑time trading, container orchestration). Therefore, a rollout must be treated as a controlled transaction with explicit pre‑checks, a defined commit boundary, verification gates, rollback triggers, bounded blast radius, and clear operator intervention points. The pattern described below enforces these execution boundaries to minimize risk while delivering the security benefit of ARP Inspection.

Designing the Smallest-Safe Rollout Pattern

Identifying Canary Scopes

The canary scope is the smallest set of devices, ports, or VLANs where ARP Inspection is first enabled. Selection criteria include:

The canary is deliberately small enough that a full‑rollback (disabling AI on the canary) can be performed within a maintenance window without affecting production services. Typical sizes: 1–2 racks, 5–10 VLANs, or ≤5% of total switch ports.

Defining Exception Handling Mechanisms

Exception handling must be explicit, deterministic, and logged. The mechanisms are:

1. Static ARP ACL Allowlist – Pre‑populated with MAC/IP pairs for known critical services (e.g., VIPs, gateway routers). Packets matching the allowlist bypass AI validation and are always forwarded.
2. DHCP Snooping Trust Boundaries – Only ports marked as dhcp snooping trust are permitted to insert new bindings; untrusted ports are subject to AI validation.
3. Rate‑Limited Logging – AI drops are logged to a local buffer and forwarded to a syslog server at a configurable rate (e.g., 100 msgs/sec) to avoid log‑spam during a mis‑configuration.
4. Fallback to Passive Mode – Some platforms support a monitor-only mode where AI logs violations but does not drop packets. This mode can be used as an intermediate verification step.

All exception handling rules are version‑controlled (e.g., stored in a Git repo) and applied via an automated configuration push that validates syntax before committing.

Establishing Stop Conditions for Legitimate Hosts

A stop condition is a measurable event that triggers an immediate halt to the rollout and initiates rollback or operator review. For ARP Inspection, the following metrics are used:

When any stop condition fires, the automation pipeline does not proceed to the next phase; it either holds the current state awaiting operator confirmation or executes a predefined rollback script.

Implementing ARP Inspection Across Environments

Bare Metal Environment Considerations

Bare‑metal servers typically connect to top‑of‑rack (ToR) switches via physical NICs. Key points:

• Link‑level consistency – Ensure NICs are configured with a static MAC address (no random MAC generation) to avoid binding churn.
• VLAN trunking – If servers trunk multiple VLANs, AI must be enabled on each VLAN individually or via a VLAN‑range command.
• Offload capabilities – Some NICs support ARP offload; verify that the switch’s AI implementation does not conflict with NIC offload (disable ARP offload on the NIC if needed).
• Power‑on sequencing – During server boot, ARP probes may occur before DHCP snooping has learned bindings; enable a short learning period (e.g., 30 s) where AI runs in monitor‑only mode.

Virtualization Environment Considerations

Virtual switches (e.g., Linux Bridge, Open vSwitch, Cisco Nexus 1000V) present additional layers:

• VM MAC mobility – Live migration can cause a MAC to appear on a different port; AI must rely on DHCP snooping bindings that are updated via the hypervisor’s DHCP relay or via static bindings injected by orchestration.
• VLAN tagging vs. VXLAN – In overlay networks, ARP inspection is performed on the underlay VLAN that carries the VXLAN traffic; ensure the underlay VLAN is AI‑enabled.
• Promiscuous mode – Some monitoring tools set NICs to promiscuous mode; AI treats these ports as untrusted and validates ARPs accordingly.
• API‑driven binding updates – Use the cloud‑provider’s metadata API or an external IPAM to push static ARP ACLs when a VM is created/deleted.

Fast-Churn Workload Environment Considerations

Containers and serverless functions generate short‑lived endpoints with rapidly changing IPs/MACs. Strategies:

• Ephemeral binding tables – Leverage DHCP snooping with a very short lease time (e.g., 30 s) and enable the switch to age out bindings quickly.
• Static ARP ACL for service IPs – For well‑known service IPs (e.g., Kubernetes service cluster IPs), install static entries in the ARP ACL so they are never subject to AI validation.
• Namespace‑level AI – On Linux‑based hosts, enable arp_ignore/arp_announce sysctls and use eBPF‑based AI (e.g., tc filter) to validate ARPs per namespace.
• Rate‑limit exemption – Exclude the container‑runtime’s internal bridge (e.g., cbr0) from AI to prevent ARP storms caused by frequent pod creation/deletion.

Configuration and Deployment

CLI Examples for ARP Inspection Configuration

Below are representative CLI snippets for common platforms. Adjust VLAN IDs, interface ranges, and ACL names to match your design.

Cisco IOS/XE (switch)

! 1. Ensure DHCP snooping is enabled globally
ip dhcp snooping
ip dhcp snooping vlan 100,101,102
ip dhcp snooping information option

! 2. Trust the uplink to the distribution layer
interface TenGigabitEthernet1/0/1
 ip dhcp snooping trust
 spanning-tree portfast trunk
!
! 3. Define static ARP ACL for critical services
arp access-list CRITICAL-SERVICES
 permit ip host 10.0.0.10 mac host 001a.2b3c.4d5e
 permit ip host 10.0.0.20 mac host 001a.2b3c.4d5f
exit
!
! 4. Apply ARP inspection on the canary VLANs
ip arp inspection vlan 100
ip arp inspection vlan 100 validate src-mac dst-mac ip
ip arp inspection vlan 100 logging
ip arp inspection filter CRITICAL-SERVICES vlan 100
!
! 5. Set aggressive logging rate-limit (optional)
ip arp inspection log-buffer logs 100 interval 1

Juniper EX Series (Junos)

[edit]
set protocols dhcp-snooping vlan 100
set protocols dhcp-snooping vlan 100 interface ge-0/0/1.0
set protocols dhcp-snooping vlan 100 interface ge-0/0/2.0 disable
set protocols arp-inspection vlan 100
set protocols arp-inspection vlan 100 validate source-mac destination-mac ip
set protocols arp-inspection vlan 100 logging
set firewall family ethernet-switching filter ARP-ALLOWLIST term allow-critical from source-address 10.0.0.10/32
set firewall family ethernet-switching filter ARP-ALLOWLIST term allow-critical from source-mac 00:1a:2b:3c:4d:5e
set firewall family ethernet-switching filter ARP-ALLOWLIST term allow-critical then accept
set protocols arp-inspection vlan 100 filter ARP-ALLOWLIST

Linux Bridge (using ebtables/arp_tables)

# Enable ARP tables module
modprobe arp_tables

# Create an ARP table for the bridge
arp -t arp_filter -N

# Add static allowlist entries
arp -t arp_filter -i br0 -s 10.0.0.10 -f 00:1a:2b:3c:4d:5e -j ACCEPT
arp -t arp_filter -i br0 -s 10.0.0.20 -f 00:1a:2b:3c:4d:5f -j ACCEPT

# Drop everything else (default policy)
arp -t arp_filter -i br0 -P DROP

# Log drops (rate‑limited via iptables)
iptables -I FORWARD -p arp --arp-op 1 -j LOG --log-prefix "ARP-DROP: " --log-level 4 --limit 10/min

Code Snippets for Automated Deployment

Ansible Playbook (Cisco NX‑OS)

- name: Deploy ARP Inspection on canary VLANs
  hosts: nxos_switches
  vars:
    canary_vlans: [100,101,102]
    critical_arp_acl: CRITICAL-SERVICES
  tasks:
    - name: Ensure DHCP snooping enabled
      nxos_config:
        lines:
          - ip dhcp snooping
          - ip dhcp snooping vlan {{ canary_vlans | join(',') }}
        before: no ip dhcp snooping
      register: dhcp_snoop

    - name: Configure static ARP ACL
      nxos_config:
        lines:
          - arp access-list {{ critical_arp_acl }}
          - permit ip host 10.0.0.10 mac host 001a.2b3c.4d5e
          - permit ip host 10.0.0.20 mac host 001a.2b3c.4d5f
        before: no arp access-list {{ critical_arp_acl }}

    - name: Enable ARP inspection on canary VLANs
      nxos_config:
        lines:
          - ip arp inspection vlan {{ item }}
          - ip arp inspection vlan {{ item }} validate src-mac dst-mac ip
          - ip arp inspection vlan {{ item }} logging
          - ip arp inspection filter {{ critical_arp_acl }} vlan {{ item }}
        loop: "{{ canary_vlans }}"

    - name: Verify ARP inspection status
      nxos_command:
        commands:
          - show ip arp inspection