Default Deny Broke Readiness Not Traffic

Introduction to Network Policies

NetworkPolicy is a Kubernetes resource that controls pod‑to‑pod communication using label selectors. It is enforced by the CNI plugin; if the plugin lacks support, the policy is accepted but has no effect.

Network policies enable least‑privilege segmentation by:

• Restricting traffic to required ports/protocols.
• Isolating workloads handling sensitive data.
• Preventing lateral movement after a compromise.
• Making service‑to‑service intent visible and version‑controlled.

Misapplied policies can allow application traffic while blocking health‑check probes, node‑local DNS, or kubelet‑originated control paths.

Understanding the Policy Case

Application traffic (e.g., HTTP from a frontend to a backend) usually originates from pods matching a permissive NetworkPolicy. Liveness/readiness probes, however, are generated by the kubelet on the node and source from the node’s IP. If a policy only allows ingress from pods with specific labels, the kubelet‑source traffic is dropped, causing probe failures even though the service works via client pods.

Node‑local DNS (e.g., CoreDNS running with hostNetwork) answers queries from pods on the same node; the response comes from the DNS pod’s IP. If the DNS pod’s egress policy only permits traffic to labeled pod IPs, the response may be blocked. Similarly, kubelet‑originated paths such as /metrics, kubectl exec, and log streaming originate from the node and are subject to the same label‑based restrictions.

Identifying the Enforcement Boundary

The CNI evaluates a packet against a NetworkPolicy by:

1. Checking if the destination pod matches the podSelector.
2. For ingress, evaluating each from rule (podSelector, namespaceSelector, ipBlock). The first matching rule allows the packet; otherwise it is denied. Egress follows the same logic with the source pod as the subject.

When the source is the node (kubelet) or a host‑networked DaemonSet, the source IP has no pod label, so podSelector and namespaceSelector do not match. Only an ipBlock that explicitly includes the node’s IP range can permit the traffic. Recognizing this gap is key to making the boundary explicit.

Troubleshooting Failed Probes and Paths

Verify Application Connectivity

# Run a temporary client pod to test the service
kubectl run test-client --rm -i --tty --image=busybox -- \
  sh -c "wget -qO- http://backend-service:8080/health && echo OK"

Check Probe Status

kubectl describe pod <backend-pod> | grep -A 6 Liveness

Capture Probe Traffic on the Node

tcpdump -i any host <backend-pod-ip> and port 8080 -w /tmp/probe.pcap

If SYN packets arrive from the node’s IP but no SYN‑ACK appears, the CNI is dropping the packet.

List Applicable NetworkPolicies

kubectl get networkpolicy -n <namespace> \
  -o jsonpath='{range .items[?(@.spec.podSelector.matchLabels.app=="backend")]}{.metadata.name}{"\n"}{end}'

Inspect a Specific Policy

kubectl get networkpolicy backend-allow-frontend -n <namespace> -o yaml

Look for missing from entries that would cover the node’s IP or the kubelet’s UID.

Example Policy to Allow Probe Traffic

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-allow-probes
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector: {}          # pods in the same namespace
    - namespaceSelector: {}    # same namespace (explicit)
    - ipBlock:
        cidr: 10.240.0.0/16    # adjust to your node CIDR
    ports:
    - protocol: TCP
      port: 8080               # application / probe port

Apply and verify:

kubectl apply -f backend-allow-probes.yaml
kubectl wait --for=condition=ready pod -l app=backend --timeout=60s

Making the Enforcement Boundary Explicit

Annotations do not affect enforcement but document intent and aid automation.

Example: Annotated Policy for Node‑Local DNS

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: dns-allow-node-local
  namespace: kube-system
  annotations:
    intent.allow.source: "node-local-dns"
    intent.allow.destination: "pod-dns-query"
    intent.allow.protocol: "UDP"
    intent.allow.port: "53"
spec:
  podSelector:
    matchLabels:
      k8s-app: kube-dns
  policyTypes:
  - Ingress
  ingress:
  - from:
    - ipBlock:
        cidr: 10.240.0.0/16   # node IPs
    ports:
    - protocol: UDP
      port: 53

Applying a Policy to a Deployment

kubectl label namespace production purpose=frontend
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-allow-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: backend
    ports:
    - protocol: TCP
      port: 8080
EOF

Verify enforcement with the CNI‑specific tool (e.g., calicoctl or cilium bpf policy get).

Scaling Limitations and Considerations

Each NetworkPolicy becomes dataplane rule evaluations. As rule count grows, latency can increase, especially with iptables‑based CNIs.

• Calico: ~1 µs per packet for < 500 rules; noticeable degradation beyond ~5k rules.
• Cilium: eBPF maps keep latency sub‑microsecond up to tens of thousands of rules.
• Kube‑router: Similar to iptables; watch for NAT table exhaustion.

Scaling tips:

• Consolidate rules using namespace selectors or broader ipBlock ranges.
• Use hierarchical policies (if supported) to avoid duplication.
• Prefer ipBlock for frequent allow‑lists instead of many podSelectors.

Example: Scaling Workers to a Shared Database

kubectl scale deployment/worker --replicas=500 -n analytics

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: workers-to-db
  namespace: analytics
spec:
  podSelector:
    matchLabels:
      role: database
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: worker
    ports:
    - protocol: TCP
      port: 5432

Verify:

kubectl get pods -l app=worker -n analytics -o wide | wc -l   # should show 500
kubectl get networkpolicy workers-to-db -n analytics -o yaml   # confirm unchanged

Monitor node dataplane CPU; sustained > 30 % suggests revisiting policy granularity.

Implementing Explicit Enforcement Boundaries via the Kubernetes API

NetworkPolicy is a namespaced resource creatable through kubectl apply, dynamic clients, or direct API calls.

Example: API JSON for Monitoring Ingress

{
  "apiVersion": "networking.k8s.io/v1",
  "kind": "NetworkPolicy",
  "metadata": {
    "name": "allow-metrics-from-monitoring",
    "namespace": "prod"
  },
  "spec": {
    "podSelector": {
      "matchLabels": {
        "app": "api"
      }
    },
    "policyTypes": ["Ingress"],
    "ingress": [
      {
        "from": [
          {
            "namespaceSelector": {
              "matchLabels": {
                "name": "monitoring"
              }
            }
          }
        ],
        "ports": [
          {
            "protocol": "TCP",
            "port": 9090
          }
        ]
      }
    ]
  }
}

Creating the Policy with curl

curl -k -X POST "$APISERVER/api/v1/namespaces/prod/networkpolicies" \
  -H "Authorization: Bearer $KUBE_TOKEN" \
  -H "Content-Type: application/json" \
  -d @- <<'EOF'
{
  "apiVersion": "networking.k8s.io/v1",
  "kind": "NetworkPolicy",
  "metadata": {
    "name": "allow-metrics-from-monitoring",
    "namespace": "prod"
  },
  "spec": {
    "podSelector": {
      "matchLabels": {
        "app": "api"
      }
    },
    "policyTypes": ["Ingress"],
    "ingress": [
      {
        "from": [
          {
            "namespaceSelector": {
              "matchLabels": {
                "name": "monitoring"
              }
            }
          }
        ],
        "ports": [
          {
            "protocol": "TCP",
            "port": 9090
          }
        ]
      }
    ]
  }
}
EOF

Verify:

kubectl get networkpolicy allow-metrics-from-monitoring -n prod -o yaml

Best Practices for Network Policy Enforcement

Monitoring and Logging

Most CNIs expose drop counters or logs:

• Calico: enable felix logging.
• Cilium: enable Hubble.
• Kube‑router: use iptables -L -v -n.

Regularly audit:

• Dropped packets per policy.
• Unexpected sources (e.g., node IPs) in drop logs.
• Policy changes via kubectl get networkpolicy -o yaml | git diff.

Example: Observing Drops with Cilium Hubble

hubble observe --type drop                     # real‑time drops
hubble observe --type drop --namespace=production --summary   # per‑namespace summary

Example: Logging with Calico (via annotation)

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: logged-allow-frontend
  annotations:
    calico/packet-log: "true"
spec:
  selector: app == 'frontend'
  ingress:
  - action: Allow
    protocol: TCP
    destination:
      ports: [8080]
    source:
      selector: app == 'backend'

Check logs:

kubectl logs -n kube-system -l k8s-app=calico-node | grep "logged-allow-frontend"

Advanced Network Policy Configurations

Overlapping Selectors

When multiple policies match a pod, the result is the union of all allowed ingress/egress rules (no “first‑match wins”). Overlap can layer intent (baseline deny‑all + specific allows) but may obscure which policy actually permitted a flow, complicating troubleshooting.

Example: Baseline Deny‑All plus Specific Allow

# deny-all-staging
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-staging
  namespace: staging
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

# allow-frontend-to-backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: staging
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
  - Egress
  egress:
  - to:
    - podSelector:
        matchLabels:
          app: backend
    ports:
    - protocol: TCP
      port: 8080

This baseline ensures only explicitly allowed traffic passes, simplifying audits.