ACL canaries with shadow counters first

#Pre-Enforcement Testing Using Shadow Rules, Counter Baselines, and Synthetic Flow Probes

Introduction

Shadow rules let you program ACL or policy entries that count matching packets without enforcing an action. Counter baselines capture packet and byte counts before a change, while synthetic flow probes generate deterministic traffic to validate the behavior of those rules. Together, they enable you to test deny logic, rule ordering, and blast‑radius impact under production‑like traffic without a cutover.

Understanding Shadow Rules

Definition and Purpose

A shadow rule mirrors the match criteria (source/destination IP, port, protocol, DSCP, VLAN, etc.) of a proposed ACL or segmentation policy but uses a non‑enforcing action such as count, log, or monitor instead of permit/deny. The rule resides in the same TCAM or flow table as regular rules, so its position reflects the true evaluation order a packet would experience. By observing which packets hit the rule under live traffic, you can uncover unintended overlaps, overly broad matches, or missing exclusions before flipping the rule to an enforcing action.

Platform‑Specific Configuration

Example CLI Commands

# Cisco IOS-XR: shadow ACL entry for denying 10.0.0.0/8 → 10.1.2.0/24 TCP 443
ipv4 access-list SHADOW_DB_DENY
 10 deny tcp 10.0.0.0 0.255.255.255 10.1.2.0 0.0.0.255 eq 443 shadow
!
# Apply the shadow ACL in the same direction as the production ACL
interface GigabitEthernet0/0/0/0
 ipv4 access-group SHADOW_DB_DENY ingress

# Juniper Junos: shadow firewall filter term
firewall {
    filter SHADOW_DB_DENY {
        term DENY_HTTPS {
            from {
                source-address {
                    10.0.0.0/8;
                }
                destination-address {
                    10.1.2.0/24;
                }
                destination-port 443;
                protocol tcp;
            }
            then {
                count SHADOW_DB_DENY_CTR;
                # no accept or discard → pure observation
            }
        }
    }
}
# Apply filter to interface
set interfaces ge-0/0/0 unit 0 family inet filter input SHADOW_DB_DENY

Implementing Counter Baselines

Role of Counter Baselines

A counter baseline records the cumulative packet and byte counts for each shadow rule (or existing production counters) over a representative measurement window—typically several minutes to an hour of live traffic. After deploying shadow rules for a proposed change, a second measurement yields a post‑shadow counter set. The delta (post‑shadow minus baseline) quantifies how many packets would have been affected by the new rule had it been enforcing. This delta is the core evidence used to decide whether to proceed, roll back, or refine the policy.

Setting Up Baselines

1. Select measurement interval – capture peak and off‑peak patterns (e.g., 5 min during business hours, 5 min during low‑usage).
2. Enable counters – ensure shadow rules have counting enabled and that the device’s polling interval aligns with the window.
3. Collect baseline – pull counters via SNMP, RESTCONF, gNMI, or CLI and store with timestamps.
4. Apply shadow rules – deploy the shadow ACL/filter/policy in the exact evaluation order.
5. Collect post‑shadow counters – repeat polling after the interval.
6. Compute delta – subtract baseline values from post‑shadow values per rule.
7. Analyze – non‑zero deltas indicate packets that would have matched the proposed action.

Code Examples

# gNMI subscription to collect ACL counters on a Cisco XR device
subscription:
  - path: "Cisco-IOS-XR-pfi-im-cmd:access-list/access-list-entry[counter]"
    mode: SAMPLE
    sample_interval: 10000000000   # 10 seconds
    heartbeat_interval: 30000000000 # 30 seconds

# CLI snippet to capture baseline counters (Cisco IOS-XR)
show access-lists ipv4 SHADOW_DB_DENY | include ^[0-9]\+
# Example output:
# 10 deny tcp 10.0.0.0/8 10.1.2.0/24 eq 443 shadow (523 packets, 45824 bytes)

# Python pseudo‑code to compute delta
baseline = {"SHADOW_DB_DENY": {"pkts": 523, "bytes": 45824}}
post_shadow = {"SHADOW_DB_DENY": {"pkts": 610, "bytes": 53456}}
delta = {
    k: {
        "pkts": post_shadow[k]["pkts"] - baseline[k]["pkts"],
        "bytes": post_shadow[k]["bytes"] - baseline[k]["bytes"]
    }
    for k in baseline
}
print(delta)  # {'SHADOW_DB_DENY': {'pkts': 87, 'bytes': 7632}}

Deploying Synthetic Flow Probes

Introduction

Synthetic flow probes are traffic generators that emit packets with precisely defined headers and payloads. They provide repeatable, controllable traffic independent of application variability, enable synchronization with measurement windows, allow testing of edge cases (malformed packets, uncommon ports), and validate deny logic by confirming that shadow counters increment as expected while no packets are dropped (since the rule remains in shadow mode).

Configuration Steps

1. Define traffic profile – source/destination IP ranges, ports, protocol, packet size distribution, inter‑packet gap (IPG).
2. Select injection point – typically a test port connected to a VLAN/VRF that mirrors the production path, or a host within the segment that can source packets toward the target.
3. Enable timestamping – embed a sequence number or timestamp in the payload to correlate with counter polls.
4. Run for the measurement interval – match the interval used for baseline collection.
5. Collect and correlate – after the test, retrieve shadow counters and verify that the expected number of probe packets matched each rule.

Example Use Cases

• Testing a new deny rule – send HTTP traffic from an untrusted subnet to a protected web server; expect the shadow deny counter to increase by the number of probe packets while actual connections succeed (because the rule is shadow).
• Validating rule ordering – generate two traffic classes that match overlapping criteria; observe which shadow counter increments first to confirm the intended evaluation order.
• Measuring burst impact – send a short burst of high‑rate packets to ensure the shadow rule’s counter does not overflow and that the device’s counter polling captures the burst accurately.
• Checking asymmetric paths – inject probes from multiple ingress points to verify that shadow counters on both ingress and egress devices reflect the same flow, exposing potential mis‑aligned ACLs.

Troubleshooting Pre-Enforcement Testing

Common Issues

• Counter not incrementing – the shadow rule may be placed after a matching permit rule, so packets never reach it. Verify the rule’s position in the ACL/filter order.
• Counter increments too high – overlapping shadow rules or bidirectional traffic can cause double‑counting. Review rule overlap and consider using directional counters.
• Baseline drift – if the measurement window captures a traffic anomaly (e.g., a backup job), the delta may be misleading. Choose stable intervals or discard outliers.
• Polling interval mismatch – counters may wrap or be missed if the polling period exceeds the counter’s roll‑over threshold. Align polling frequency with expected traffic rates and counter size.

By following the practices above, you can safely stage ACL or segmentation changes, validate deny logic under realistic traffic, and reduce the risk of service disruption during enforcement.