NetworkPolicy intent versus real enforcement after encapsulation

Introduction to Service-to-Service Policy

Service‑to‑service (S2S) policy expresses the intended communication relationships between workloads, typically as allow/deny rules that reference logical identities (pod labels, service names, security groups, workload tags). Authored declaratively (YAML, JSON, DSL) and stored in a source‑of‑truth repository, the intended state answers: Which logical endpoints may initiate which transport‑level flows, and on which ports/protocols?

Example Calico NetworkPolicy:

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-db
spec:
  selector: has(label='frontend')
  ingress:
    - action: Allow
      protocol: TCP
      destination:
        ports: [5432]
        selector: has(label='postgres')

This states: any workload labeled frontend may open a TCP connection to port 5432 on any workload labeled postgres. No other traffic is permitted.

Importance of Service Segmentation

Segmentation limits the blast radius of a compromised workload. Enforcing S2S policy at the network layer prevents lateral movement unless explicitly allowed and supports compliance regimes (PCI‑DSS, HIPAA) that require isolation of sensitive data flows. When policy is correctly rendered, applied, and observed, the observed state matches the intended state: only allowed flows appear in flow logs or packet captures.

Understanding Hook Points and Headers

Inner vs. Outer Headers in Overlay Traffic

Overlay encapsulation (VXLAN, Geneve, GRE, etc.) adds an outer header carrying underlay transport addresses (typically host NIC IPs) and an inner header preserving the original packet’s source/destination addresses, ports, and protocol.

A hook point is any location in the packet processing path where a policy decision can be made. Depending on its location, it may see only the outer header, only the inner header, or both (after decapsulation).

• Pre‑encapsulation hook (e.g., eBPF on a veth before the vxlan device): sees inner header only.
• Post‑decapsulation hook (e.g., netfilter INPUT chain on the host after the vxlan device): sees inner header after outer header stripped.
• Underlay‑only hook (e.g., physical switch ACL, NIC offload filter, underlay‑based security group): sees only the outer header.

If a policy engine enforces based on inner‑header identities but the actual enforcement point is an underlay‑only hook, the policy is evaluated on the wrong address space, causing a mismatch between intended and observed state.

How Hook Points Interact with Traffic

In a Kubernetes cluster running Calico in VXLAN mode, the data path is:

1. Pod A → veth pair → Linux bridge → vxlan device (encapsulation) → physical NIC → underlay network → physical NIC of host B → vxlan device (decapsulation) → bridge → veth → Pod B.

Policy enforcement points can be placed at:

• Ingress/egress of the veth (pre‑encap) – sees inner header.
• Ingress/egress of the vxlan device (post‑encap/pre‑decaps) – sees outer header only (if the device does not decaps before the hook).
• Netfilter FORWARD chain on the bridge (post‑decaps) – sees inner header.
• Physical switch ACL – sees outer header only.

The rendered state is the set of low‑level rules generated by the policy controller for each hook point. The applied state is what the kernel or hardware actually has loaded (e.g., iptables -L, tc filter show, switch show access-list). The observed state is derived from telemetry (flow logs, packet captures, eBPF tracepoints) showing which packets were permitted or dropped.

Policy Enforcement and Overlay Traffic

Intended Policy vs. Actual Enforcement

The intended policy is expressed in terms of overlay identities (pod IPs, labels). Actual enforcement depends on which hook point the policy controller targets. Misalignment occurs when:

1. The controller renders rules for an inner‑header hook (e.g., calico-node iptables rules on the veth) but the enforcement point is an outer‑header hook (e.g., a physical switch ACL programmed via an orchestrator that only sees underlay IPs).
2. The controller renders rules for an outer‑header hook (e.g., NSX Distributed Firewall rules on the uplink) but the observed traffic is measured after decapsulation, leading to a false sense of compliance.

In both cases, the rendered state may look correct (the policy compiler emitted the expected rules), but the applied state at the actual enforcement boundary diverges, causing the observed state to violate the intended segmentation.

Scenarios Where Overlay Traffic Violates Segmentation Intent

In each case, the intended state (allow/deny based on overlay identities) is satisfied on paper because the policy compiler emitted the correct rules. However, the applied state at the actual enforcement point (outer‑header only) leads to an observed state where traffic that should be denied is permitted, or vice‑versa.

Troubleshooting Policy Enforcement Issues

Identifying Mismatches Between Intended and Actual Policy

1. Collect the intended policy from the source of truth (e.g., kubectl get networkpolicy -o yaml).
2. Determine the enforcement point(s) used by the CNI/plug‑in:
   • Calico: calicoctl get node <node> -o yaml shows felixConfiguration and iptables vs ebpf mode.
   • Cisco ACI: inspect the applied contract (vzBrCP) and the associated EPG’s pcTag.
   • NSX: view the DFW section (GET /api/v1/dfw/rules).
3. Dump the applied state at each hook:
   • iptables -S -t filter (or nft list ruleset) for netfilter/ebpf.
   • tc filter show dev veth0 for eBPF/XDP.
   • ovs-ofctl dump-flows br-int for OVS‑based encap/decap.
   • show access-list on the physical switch.
4. Capture traffic at strategic points to see which headers are present:
   • tcpdump -i veth0 -nn -s0 -w pod-to-pod.pcap (inner).
   • tcpdump -i eth0 -nn -s0 -w underlay.pcap (outer).
   • tcpdump -i vxlan0 to see encapsulated packets.
5. Compare the observed 5‑tuple (srcIP, dstIP, srcPort, dstPort, proto) from the packet capture with the rule match criteria. If the observed tuple matches a rule that only inspects outer headers, you have identified the mismatch.

Tools and Techniques for Analyzing Hook Points and Headers

• eBPF tracing: bpftrace -e 'tracepoint:net:net_dev_queue { printf("dir=%s if=%s dport=%d\\n", args->direction, args->name, ntohs(args->dst_port)) }' to see which packets hit a given hook.
• Flow logs: Enable VPC flow logs (AWS), NSX DFW logs, or Azure Network Watcher to capture the underlay 5‑tuple; compare with VPC flow logs that capture the inner 5‑tuple via VM‑level logging.
• Service mesh telemetry: Istio Mixer/Telemetry v2 can report destination.ip and destination.port as seen by the sidecar (inner). Contrast with the access log from the ingress gateway (outer).
• Policy validation frameworks: conftest with OPA policies to verify that rendered rules contain the expected selectors; kube-score for NetworkPolicy linting.
• Diffing intended vs applied: Export iptables rules to JSON (iptables-save | iptables-restore -t) and compare with the rendered policy using diff -u.

Code and CLI Examples for Policy Configuration

Configuring Service-to-Service Policy Using CLI

Calico (iptables mode)

# Create a NetworkPolicy that allows frontend → postgres
cat <<EOF | kubectl apply -f -
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-postgres
spec:
  selector: has(label='frontend')
  ingress:
    - action: Allow
      protocol: TCP
      destination:
        ports: [5432]
        selector: has(label='postgres')
EOF

Verify rendered iptables rules on a node:

# Felix renders rules into chains like cali‑from‑wl‑endpoint
sudo iptables -S | grep -A5 -B5 "cali‑from‑wl‑endpoint"

Cisco ACI (REST API)

# Create a contract that permits web → db on TCP 8080
curl -k -u admin:password \
  -X POST https://apic/api/mo/uni/tn-Tenant1/brc-WebToDb.json \
  -d '{
        "vzBrCP": {
          "attributes": {
            "name": "WebToDb",
            "scope": "context"
          },
          "children": [
            {
              "vzSubj": {
                "attributes": {
                  "name": "allowWebDb",
                  "prot": "tcp"
                },
                "children": [
                  {
                    "vzRsSubjFiltAtt": {
                      "attributes": {
                        "tnVzFilterName": "http-8080"
                      }
                    }
                  }
                ]
              }
            }
          ]
        }
      }'

Apply the contract to EPGs:

# Consume/provide
curl -k -u admin:password \
  -X POST https://apic/api/mo/uni/tn-Tenant1/app-WebApp/epg-Web.json \
  -d '{
        "fvAEPg": {
          "attributes": {
            "name": "Web",
            "pcEnfPref": "enabled"
          },
          "children": [
            {
              "vzRsProv": {
                "attributes": {
                  "tnVzBrCPName": "WebToDb"
                }
              }
            },
            {
              "vzRsCons": {
                "attributes": {
                  "tnVzBrCPName": "WebToDb"
                }
              }
            }
          ]
        }
      }'

Verify applied state on the leaf:

# Show the contract hit count
show access-list | include WebToDb
show np count | include WebToDb

NSX-T (Distributed Firewall)

# Create a DFW section via API
curl -u admin:password -k \
  -X POST https://nsx-manager/api/v1/dfw/sections \
  -H "Content-Type: application/json" \
  -d '{
        "section_id": "frontend-to-db",
        "display_name": "Allow frontend → db TCP/5432",
        "applied_to": [
          {"target_type": "IPSet", "target_id": "ipset-frontend"},
          {"target_type": "IPSet", "target_id": "ipset-db"}
        ],
        "rules": [
          {
            "action": "ALLOW",
            "protocol": "TCP",
            "destination_ports": ["5432"],
            "direction": "INOUT"
          }
        ]
      }'

Verify applied rules on a transport node:

# Get the rule set as seen by the host
vcli get dfwrule -section-id frontend-to-db

Example Code for Verifying Hook Points and Headers

Python + Scapy to sniff inner vs outer

from scapy.all import sniff, IP, UDP, TCP

def pkt_cb(pkt):
    if UDP in pkt and pkt[UDP].dport == 4789:  # VXLAN port
        print("Outer header:", pkt[IP].src, "->", pkt[IP].dst)
        if IP in pkt:  # inner IP after UDP
            print("Inner header:", pkt[IP].src, "->", pkt[IP].dst)
            if TCP in pkt:
                print("Inner TCP sport/dport:", pkt[TCP].sport, "->", pkt[TCP].dport)

sniff(iface="eth0", prn=pkt_cb, store=0)