Calico precedence traps during cross-zone canaries

Introduction to Canary Deployments and Calico Policies

Overview of Canary Deployments

Canary deployments release a new version of a service to a small subset of traffic or nodes before a full rollout. In Kubernetes, this is often achieved by labeling a subset of pods (the canary) with a distinct label or annotation and routing a fraction of requests to them via a Service, Ingress, or service-mesh traffic split. The goal is to validate functional correctness, performance, and security posture in a production-like environment while limiting blast-radius.

Calico Network Policies and Their Role in Kubernetes

Project Calico provides a CNI plugin that enforces network segmentation through Calico network policy objects. These objects are CRDs that map to iptables/eBPF rules on each node. Calico distinguishes two policy scopes:

Understanding the Issue: Canary Failures Across Zones

Identifying the Symptoms

In a multi-zone Kubernetes cluster (e.g., three AZs: us-east-1a, us-east-1b, us-east-1c), a canary deployment of service frontend exhibits the following pattern:

• Traffic from the backend service to canary pods succeeds when the canary pods are scheduled in zone a but fails (connection timed out / TCP reset) when the same canary pods are scheduled in zones b or c.
• The backend pods themselves are spread evenly across all zones and show no intra-zone communication problems.
• No application-level errors are logged; the failure appears at the TCP SYN-ACK stage.

Initial Troubleshooting Steps

1. Verify pod scheduling – kubectl get pods -l version=canary -o wide confirms canary pods are present in all zones.
2. Check Service endpoints – kubectl get endpoints backend shows endpoints spread across zones; the Service’s clusterIP is reachable from all zones.
3. Capture packets – tcpdump on a backend node shows SYN packets sent to the canary pod IP, but no SYN-ACK returned when the pod resides in zones b or c.
4. Inspect Calico policy enforcement – calicoctl get globalnetworkpolicy -o yaml and calicoctl get networkpolicy -n <ns> -o yaml reveal the set of rules applied to the backend and canary pods.

Calico Global Policy and Its Application

Definition and Configuration of Global Policies

A globalnetworkpolicy is a cluster-scoped CRD that is not bound to any namespace. It is useful for baseline deny-all or allow-specific rules that should apply uniformly. The policy is evaluated in the tier specified by its spec.order (lower numbers evaluated first) or, if omitted, in the default calico tier. Key fields:

• spec.selector – pod selector (e.g., has(app.kubernetes.io/name)).
• spec.namespaceSelector – namespace selector (optional; if omitted, matches all namespaces).
• spec.serviceAccountSelector – service account selector (optional).
• spec.ingress / spec.egress – list of allowed rules.
• spec.order – integer controlling evaluation order within a tier.
• spec.tier – name of the tier; if omitted, uses the default tier.

Examples of Global Policy Implementation

# Create a tier named "baseline" with order 100 (evaluated before default tier)
cat <<EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: Tier
metadata:
  name: baseline
spec:
  order: 100
EOF

# Global deny-all except for a specific service account in the "frontend" namespace
cat <<EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: deny-all-except-frontend-sa
spec:
  tier: baseline
  order: 50
  selector: has(serviceaccount)
  types: [Ingress, Egress]
  ingress:
  - action: Allow
    source:
      serviceAccount:
        name: frontend-sa
        namespace: frontend
  egress:
  - action: Allow
    destination:
      selector: has(serviceaccount)
EOF

Namespace Policy and Its Interaction with Global Policy

Namespace Policy Definition and Configuration

A namespaced networkpolicy lives inside a Kubernetes namespace and only matches pods in that namespace. It follows the same selector syntax as global policies but is scoped implicitly to its namespace. The effective policy for a pod is the union of all matching global and namespaced policies, ordered by tier → order → specificity.

Precedence Order: Global vs. Namespace Policies

Calico evaluates policies in the following sequence:

1. Tier order – tiers are sorted by spec.order (ascending).
2. Order within tier – each policy’s spec.order (ascending).
3. Policy type – within the same order, globalnetworkpolicy is evaluated before networkpolicy.
4. Selector specificity – if two policies have identical tier and order, the one with the more specific selector wins for the conflicting action.

Service-Account Selectors in Calico Policies

Introduction to Service-Account Selectors

Calico policies can reference Kubernetes service accounts via the serviceAccount field under source or destination selectors. The selector matches the service account object (not the pod’s annotations). This enables fine-grained segmentation based on identity rather than just pod labels.

Configuring Service-Account Selectors in Policies

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-backend-to-canary-via-sa
spec:
  tier: baseline
  order: 150
  selector: version == 'canary'
  namespaceSelector: has(kubernetes.io/metadata.name)
  serviceAccountSelector:
    name: frontend-sa
    namespace: frontend
  types: [Ingress]
  ingress:
  - action: Allow
    source:
      serviceAccount:
        name: backend-sa
        namespace: backend
    destination:
      ports:
      - 80

Reconstructing the Evaluation Path for Consistent Policy Application

Step-by-Step Policy Evaluation Process

1. Packet arrives at the veth interface of the source pod (backend).
2. Calico dataplane extracts:
   • Source pod labels, namespace, service account.
   • Destination pod IP → resolves to destination pod’s labels, namespace, service account.
3. Tier list is read from the Calico dataplane (ordered by spec.order).
4. For each tier, policies are sorted by spec.order.
5. Each policy is evaluated:
   • Check selector against source pod labels (for ingress) or destination pod labels (for egress).
   • Check namespaceSelector against source/destination namespace labels.
   • Check serviceAccountSelector against source/destination service account objects.
   • If all selectors match, the rule’s action (Allow/Deny) is applied; processing stops for that direction if the action is Deny or continues if the action is Allow.
6. If no policy matches, the default action of the tier (usually Deny in the baseline tier) applies.