The remote SSH lockout after an atomic replace

Blameless Post‑Mortem: Self‑Inflicted Firewall Outage

What a blameless post‑mortem is

A structured review that answers what happened, why it happened, and how to prevent recurrence without assigning individual fault. It surfaces systemic gaps—broken assumptions, missing safety rails, detection or response deficiencies—while preserving a psychologically safe environment for engineers.

Why it matters in infrastructure‑heavy environments

• Encourages full disclosure → richer root‑cause analysis.
• Highlights missing safeguards (pre‑change validation, automated rollback) that stay hidden in blame‑centric cultures.
• Drives concrete improvements to change‑management pipelines, verification gates, and blast‑radius controls.
• Aligns with SRE/DevOps principles: treat failure as a learning opportunity.

Incident Overview

Technical details

• OS/Kernel: Linux 5.15, nftables v1.0.4.

• Change mechanism: ssh root@fwX "nft -f /opt/nftables/production.nft"

• Problematic script excerpt

  flush table inet filter
  delete table inet filter
  add table inet filter
  add chain inet filter input { type filter hook input priority 0 \; policy drop \; }
  add chain inet filter forward { type filter hook forward priority 0 \; policy drop \; }
  add chain inet filter output { type filter hook output priority 0 \; policy accept \; }
  # Intended to only add SSH rule
  add rule inet filter input tcp dport 22 ip saddr 10.0.0.0/8 accept

• Critical flaw: The script began with flush then delete, removing the existing inet filter table and its implicit ct state established,related accept rule. The newly created input chain defaulted to drop, so all established traffic—including the SSH sessions used to push the change—was dropped.

• Blast radius: 100 % of the firewall fleet lost in‑band management; downstream load‑balancer health checks failed, causing brief service degradation.

Initial response & containment

1. Detection – Prometheus alerts (ssh_down, bgp_peer_down) fired within 20 s.
2. Isolation – Engineers switched to out‑of‑band IPMI consoles.
3. Rollback – Re‑applied the last known good ruleset from a nightly Git‑backed backup via nft -f.
4. Verification – Diffed nft list ruleset against the backup, confirmed equality, then validated SSH and BGP.
5. Communication – Incident channel updated every 5 min; a temporary “firewall‑maintenance” banner warned ops teams.

Broken Assumption

Assumption: The CI‑driven nft -f push would be purely additive—only inserting the SSH allow rule while leaving the existing rule set intact.

Basis:

• Prior experience with iptables -A style scripts that append.
• Belief that nft -f behaved similarly when the script lacked explicit delete/flush.
• No code review that would have caught the destructive lines.

How it caused the outage

1. flush table inet filter + delete table inet filter removed the existing table and all its chains.
2. A new table was created with a default drop policy on the input chain.
3. Only a single SSH allow rule was added; it matched new SYN packets from 10.0.0.0/8 but not existing ESTABLISHED traffic.
4. Established sessions (including the SSH channel used to push the change) were dropped instantly.

Lessons

• Never assume additive behavior for a tool that can replace state; verify exact semantics.
• Treat any flush, delete, or table recreation as a destructive operation requiring explicit approval.
• Enforce a “no‑destructive‑change‑without‑review” rule in CI (block scripts containing flush/delete unless a special flag is present).
• Validate the full resulting ruleset, not just the intended delta, before committing to production.

Missing Safety Rails

Key missing rails

1. Pre‑change syntactic & semantic validation (nft -c + diff‑based dry‑run).
2. Change‑scope limitation (canary hosts, feature flags, or time‑bounded windows).
3. Automated verification (connectivity test, traffic‑plane validation, BGP peer state).
4. Automatic rollback on verification failure.
5. Blast‑radius controls and an explicit operator abort point.

Safer Remote‑Change Pattern for nftables

Adopt a transaction‑oriented workflow that validates, limits exposure, and can roll back automatically.

1. Prepare a immutable backup

# On each target firewall (or via a central orchestration tool)
nft list ruleset > /var/backups/nftables/$(date +%F_%T).nft

2. Validate the proposed changeset locally

# Syntax check
nft -c -f /opt/nftables/proposed.nft

# Dry‑run: load into a temporary table and diff against current state
nft -f /opt/nftables/proposed.nft -I netdev   # load into a temporary namespace if supported
nft list ruleset > /tmp/proposed.nft
diff -u /var/backups/nftables/current.nft /tmp/proposed.nft

If the diff shows only the intended additions, proceed; otherwise abort.

3. Canary rollout

• Apply the changeset to a small subset (e.g., 1‑2 firewalls) using the same nft -f command.
• Run automated verification on the canary hosts:

  # Example verification script
  ssh fw-canary "nft list ruleset | grep -q 'tcp dport 22 accept' && \
                 ping -c 3 8.8.8.8 && \
                 bgpctl show summary | grep -q 'Established'"

• If verification fails, trigger an automatic rollback on the canary hosts (see step 5) and abort the fleet‑wide push.

4. Fleet‑wide apply with transaction boundaries

Use nft’s built‑in transaction semantics: the -f flag loads the entire file as a single atomic transaction.

ssh root@fwX "nft -f /opt/nftables/proposed.nft"

Because the load is atomic, either the whole ruleset replaces the old one or nothing changes if an error occurs.

5. Automatic rollback trigger

• After applying, run the same verification script on each host.
• If verification fails, execute:

  ssh root@fwX "nft -f /var/backups/nftables/$(ls -t /var/backups/nftables/ | head -1)"

• Orchestration tools (Ansible, Salt, etc.) can wrap steps 2‑5 in a playbook that aborts on the first verification failure and runs the rollback task.

6. Post‑change validation & cleanup

• Diff the final ruleset against the backup to confirm no drift.
• Retain the backup for a defined retention period (e.g., 30 days) for audit purposes.
• Update documentation and CI lint rules to flag any flush/delete statements unless explicitly whitelisted.

Summary of improvements

• Atomic transactions via nft -f guarantee a clean all‑or‑nothing apply.
• Pre‑change validation (nft -c + diff) catches destructive operations before they reach devices.
• Canary rollout limits blast radius and provides early feedback.
• Automated verification replaces manual SSH checks with scripted, repeatable tests.
• Automatic rollback on verification failure eliminates manual MTTR delays.
• Explicit backup/restore gives a known‑good restore point and audit trail.

Implementing this pattern transforms a risky, additive‑assumption‑driven push into a safe, verifiable, and self‑healing change process for nftables‑based firewalls.