The regex route-map that matched the whole customer table

Incident Overview

On 2024‑09‑12 at 09:14 UTC, the BGP routing table on the edge routers of AS 65001 swelled to ≈ 1.2 M customer prefixes—a 40× increase over the expected ≈ 30 k. The surge caused:

• CPU utilization peaking at 85 % on the route‑processor
• TCAM exhaustion on line cards, dropping ≈ 2 % of traffic
• Multiple BGP session resets from hold‑timer expiration
• Intermittent loss of reachability to ~15 % of hosted services for ~45 minutes until the offending route‑map was withdrawn

The root cause was an overly permissive regular expression in a route‑map that matched far more AS‑PATH strings than intended, causing the router to accept and re‑advertise every prefix learned from the upstream peer (AS 65000).

Incident Timeline

Root Cause Analysis

Intended Route‑Map Logic

The route‑map was meant to accept only customer‑originated prefixes whose AS‑PATH ends with the customer AS number and contains the peer AS 65000 exactly once:

match as-path regex "^_65000_[0-9]+$"

• _ = AS‑PATH separator (space or start/end of string)
• Expected matches: 65000 65010 (peer → customer)
• Expected rejects: transit paths, multiple peer occurrences, missing customer AS, etc.

Deployed (Incorrect) Regex

The version pushed to production was:

match as-path regex "^_65000_.*"

• .* matches any sequence after _65000_, including an empty string.
• Consequently, any AS‑PATH containing the substring _65000_ satisfies the match, regardless of what follows.

Examples of Unintended Matches

Impact of the Incorrect Regex

• The match became “accept if the AS‑PATH contains the peer AS anywhere”.
• AS 65000 advertises the full Internet table (~ 900 k prefixes) to AS 65001.
• All those prefixes were installed into the local BGP table and re‑advertised downstream.

Results

• BGP table inflated from ~30 k (expected) to ~1.2 M (full Internet + customer routes).
• CPU spikes from repeated best‑path calculations and route‑map evaluation on every UPDATE.
• TCAM exhaustion → packet drops for traffic needing longest‑prefix lookups beyond allocated depth.
• Increased update churn triggered hold‑timer expirations on several eBGP sessions, causing brief flaps.

Weak Review Gates

Code Review Process

• The change was a single Jinja2 fragment (route-maps.j2) inside a larger network‑policy repo.
• PR description: “Add route‑map to filter customer prefixes” – no mention of the regex or its intent.
• Review checklist used:
  • Syntax validation (ansible-lint)
  • No trailing whitespace
  • Commit message format
  • Verify regex matches only intended AS‑PATH patterns
  • Confirm impact analysis (prefix‑count estimate)
  • Check for over‑broad matches (.*, .+)

Approval and Validation Procedures

• Approval required two “LGTM”s from any network engineer; no senior‑engineer sign‑off mandated for routing policy changes.
• The PR received two LGTMs from junior engineers who focused on formatting and did not run show route-map RM-CUST-IN test to validate matches.
• No automated policy‑validation step (e.g., Batfish, Cisco PyATS) was integrated into the CI pipeline at the time.

Inadequate Peer Review

• Reviewers did not consult the existing unit‑test suite (see Weak Test Coverage) to see that it only covered a single positive case and omitted negative cases.
• No discussion of the regex’s anchoring (^ and $) or the meaning of .*.
• Absence of a “regex‑review” guideline meant reviewers relied on intuition rather than a concrete validation matrix.

Weak Test Coverage

Test Environment and Methodology

• Unit tests written in Python using pyATS, invoking FRR’s vtysh to load a temporary configuration and executing show route-map RM-CUST-IN test <aspath> against predefined AS‑PATH strings.
• Suite lived in tests/test_route_map_customer.py and ran via GitHub Actions on every push.
• FRR version in test container: 8.4, matching production.

Inadequate Test Scenarios

Positive test (only case covered):

assert test_aspath("65000 65010") == True   # expected match

Missing negative tests:

• 65000 65010 65020 → should be False

• 65010 65000 → peer AS not leading

• 65000 → no customer AS

• 65000 65010 65000 → peer AS appears twice

• Random Internet AS‑PATHs (e.g., 3356 1299 65000 209)

• No test for anchoring behavior (^ and $); the test harness only checked the first 20 characters of output, masking the effect of .*.

• Code coverage reported by coverage.py: 78 % lines, 42 % branches.

• The match as-path regex block had one statement, covered only by the positive case; all negative‑match branches and anchoring verification were untested.

The CI pipeline marked the build green because the sole assertion passed; the lack of negative assertions meant the faulty regex was never detected.

Slow Detection and Response

Monitoring and Alerting Mechanisms

• BGP prefix count monitored via SNMP OID 1.3.6.1.2.1.15.13.1.1.4 (bgpPrefixes) collected every 30 s into Prometheus.

• Alert rule:

  - alert: BgpPrefixCountHigh
    expr: bgp_prefixes{job="frr"} > 100000
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "BGP prefix count high on {{ $labels.instance }}"
      description: "Prefix count is {{ $value }} (> 100k)."

• No alert for rate of increase (e.g., > 10k prefixes/min).

• No TCAM utilization alerts enabled on line cards (platform‑specific SNMP OIDs omitted).

Anomaly Detection and Response Times

• BgpPrefixCountHigh warning fired at 09:09 UTC, 7 minutes after the route‑map was applied.
• NOC engineer’s first manual check (show ip bgp summary) at 09:12 UTC (3 minutes after the alert).
• Root‑cause identification (regex inspection) completed by 09:18 UTC (≈ 6 minutes after the manual check).
• Total detection‑to‑mitigation window: ≈ 18 minutes from change application to workaround.

Contributing factors

• Alert threshold set too high (100 k); normal variance is 30‑50 k, so the jump to 112 k only just crossed the threshold.
• No trend‑based alerting; reliance on a static absolute limit.
• Engineers occupied with a separate change window, delaying response.

Communication Breakdowns

• Change‑request ticket CR‑2024‑0912‑01 did not link to the network‑policy PR; approver saw only a high‑level description.
• During the incident, the NOC engineer attempted to contact the automation team via Slack but missed the threaded discussion because the notification was buried under unrelated alerts.
• No post‑change “smoke‑test” notification sent to the routing team; the team learned of the issue only through the monitoring alert.

Troubleshooting and Debugging

Initial Troubleshooting Steps

1. Verify BGP table size

   show ip bgp summary | include Prefix

   Output: 1,215,432 prefixes (expected ~30 k).

2. Check route‑map hit counters

   show route-map RM-CUST-IN

   Output: policy routing matches: 1,198,742 packets.

3. Examine a sample UPDATE

   debug bgp updates in   # filtered to a single neighbor

   Showed every UPDATE from the peer being accepted.

4. Test the regex directly

   show route-map RM-CUST-IN test as-path 65000 65010
   show route-map RM-CUST-IN test as-path 65000 65010 65020

   Both returned match, confirming the over‑broad pattern.

Debugging Techniques and Tools

• FRR route‑map test command – offline validation of a regex against arbitrary AS‑PATH strings without affecting live traffic.
• bgpdt (BGP debug trace) – captured raw UPDATEs to confirm offending AS‑PATHs in the peer’s advertisements.
• tcpdump on BGP port 179 – verified no malformed packets; the issue was purely logical.
• Prometheus query – rate(bgp_prefixes_total[5m]) showed a spike of ~ 150k prefixes/min at 09:07, correlating with the UPDATE burst.

Example CLI Commands for Debugging

# 1. Confirm prefix explosion
show ip bgp summary | awk '/Prefix/ {print $2}'

# 2. Show route-map statistics
show route-map RM-CUST-IN

# 3. Test specific AS-paths
show route-map RM-CUST-IN test as-path 65000 65010
show route-map RM-CUST-IN test as-path 65000 65010 65020
show route-map RM-CUST-IN test as-path 65000
show route-map RM-CUST-IN test as-path 65010 65000

End of review.