Staged segmentation changes with permit shadowing

Introduction to Service-to-Service Policy Changes

Overview of Permit Shadowing

Permit shadowing introduces a new allow rule in parallel with the existing policy set but marks it as non‑enforcing (shadow mode or dry‑run). The rule is evaluated by the policy engine, its decision is logged, yet the final allow/deny verdict continues to come from the authoritative rule set. This lets operators see whether the proposed rule would have permitted traffic that is currently blocked, without changing enforcement.

Importance of Negative Tests and Dependency Probes

Negative tests deliberately attempt traffic that should be denied after a policy change, verifying that the new deny line works. Dependency probes are lightweight, synthetic requests sent from a service to its known upstream/downstream partners to confirm that required communication paths remain open. Together they provide a safety net: permit shadowing shows what would be allowed, negative tests confirm what is now blocked, and dependency probes guard against unintended breakage of legitimate flows.

Understanding Permit Shadowing

Definition and Purpose

Permit shadowing creates a shadow copy of a candidate allow rule that runs in observation mode. The purpose is to gather empirical data on the rule’s match set—source selectors, destination selectors, ports, protocols, and any identity attributes—before promoting it to the enforcing policy layer. This reduces the risk of over‑permissive changes that could expand the blast radius.

Implementation Considerations

• Policy Engine Support: The enforcement point (e.g., Istio AuthorizationPolicy, Calico NetworkPolicy, OPA, AWS Security Groups) must support a dry‑run or shadow mode flag. If not, deploy a parallel policy set with higher priority but a log-only action.
• Evaluation Order: Shadow rules are evaluated after the authoritative set; their decision does not affect the final verdict. Logging must capture the rule’s match and the engine’s internal allow/deny recommendation.
• Data Retention: Retain shadow logs long enough to capture periodic and bursty traffic patterns (typically 24‑48 h for micro‑services, longer for batch workloads).
• Selector Granularity: Overly broad selectors (e.g., namespace: *) generate noise and can overwhelm logging systems; start with tight selectors and broaden only after validation.
• Rollback Path: Because shadow rules do not enforce, removing them is a no‑op; however, clean up any associated logging side‑cars or metric exporters to avoid storage bloat.

Example Configuration and Code Snippets

Istio AuthorizationPolicy (shadow mode)

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: permit-shadow-orders-to-inventory
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: inventory
  action: ALLOW   # evaluated but not enforced due to `mode: SHADOW`
  mode: SHADOW    # Istio 1.19+ feature; logs match decisions
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/orders/sa/orders-sa"]
    to:
    - operation:
        paths: ["/inventory/*"]
        methods: ["GET", "POST"]

When applied, Istio emits a log entry similar to:

[2025-11-02T14:03:12Z] authorization_policy_match: policy="permit-shadow-orders-to-inventory" result=ALLOW source="orders-sa" destination="inventory" path="/inventory/item/42"

Calico NetworkPolicy (log‑only via applyOnForward: false)

apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: deny-shadow-orders-to-payments
  namespace: default
spec:
  selector: app == 'payments'
  types: [Ingress]
  ingress:
  - action: Allow       # logged but not enforced because applyOnForward: false
    source:
      selector: app == 'orders'
    destination:
      ports: [8080]
  applyOnForward: false   # Calico 3.22+; makes rule log‑only

The policy engine increments the kpa counters for matches without dropping packets.

OPA Shadow Policy (using data.system.auth.allow with tracing)

package system.auth

# Shadow rule – logs match but does not influence final decision
shadow_allow {
    input.method == "POST"
    input.path[0] == "services"
    input.path[1] == "orders"
    input.user.roles[_] == "service:orders"
    # Emit a trace event
    trace("shadow_allow_matched", {"user": input.user.id})
}

# Authoritative deny – blocks everything else
allow {
    not shadow_allow   # if shadow didn't match, fall back to deny
    false
}

When OPA is run with --decision-logs, each evaluation shows whether shadow_allow fired.

Designing Negative Tests

Identifying Critical Dependencies

1. Service Dependency Graph: Generate a directed graph from service mesh telemetry (e.g., Istio’s istioctl proxy-config routes, or CNI flow logs) to map which services call which endpoints.
2. Failure Domains: Identify calls that cross trust boundaries (e.g., from a public‑facing ingress to an internal database) or that involve privileged service accounts.
3. Change Impact Analysis: For each candidate deny rule, list the exact source/destination selector pairs it will block. Those pairs become the negative test targets.

Creating Effective Test Scenarios

• Deterministic Requests: Use a client that can set exact headers, mTLS certificates, JWT claims, and HTTP method/path to match the rule’s selectors.
• State Isolation: Run the test client in a namespace or VPC with no overlapping allow rules that could mask the deny.
• Timing: Execute tests during low‑traffic windows to reduce noise, but also schedule periodic re‑runs to catch dynamic policy changes (e.g., autoscaling label updates).
• Assertions: Expect either a TCP reset/ICMP unreachable, an HTTP 403/429, or a policy‑engine‑generated deny log. Record both network‑level and policy‑level outcomes.

Example Negative Test (Bash + curl) for an Istio deny rule

#!/usr/bin/env bash
NAMESPACE="orders"
TARGET="inventory.orders.svc.cluster.local:8080"
PATH="/inventory/admin"
TOKEN=$(kubectl -n "$NAMESPACE" get secret orders-sa-token -o jsonpath='{.data.token}' | base64 --decode)

# Attempt a call that should be denied after the policy is enforced
STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
  --cacert /etc/istio/certs/root-cert.pem \
  --cert   /etc/istio/certs/cert-chain.pem \
  --key    /etc/istio/certs/key.pem \
  -H "Authorization: Bearer $TOKEN" \
  -X POST https://"$TARGET"$PATH)

if [[ "$STATUS" -eq 200 ]]; then
  echo "ERROR: Expected deny, got $STATUS"
  exit 1
else
  echo "OK: Received $STATUS as expected"
fi

Integrating Negative Tests into CI/CD Pipelines

1. Pipeline Stage: Add a policy-validation stage after render-manifests but before apply.
2. Test Harness: Use a container image containing the test client, required certificates, and a test runner (e.g., pytest with the requests library).
3. Result Gating: The stage fails if any negative test receives an allowed response or if the policy engine’s shadow logs show an unexpected ALLOW for the test traffic.
4. Artifact Publishing: Publish test logs and policy decision traces as pipeline artifacts for audit.
5. Canary Promotion: Only promote the policy to the production cluster if the pipeline passes on a staging cluster that mirrors production selectors.

GitHub Actions snippet

name: Service-to-Service Policy Rollout

on:
  push:
    branches: [main]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Render manifests
        run: |
          kustomize build ./overlays/staging > rendered.yaml
      - name: Apply to staging cluster
        uses: azure/k8s-set-context@v2
        with:
          method: kubeconfig
          kubeconfig: ${{ secrets.STAGING_KUBECONFIG }}
      - run: kubectl apply -f rendered.yaml
      - name: Run negative tests
        uses: docker://myorg/policy-test-runner:latest
        env:
          KUBECONFIG: ${{ runner.temp }}/kubeconfig
        with:
          args: --test-dir ./negative-tests

Implementing Dependency Probes

Types of Probes and Their Applications

Configuring Probes for Service Dependencies

In Kubernetes, probes are defined in the pod spec. For service‑to‑service validation we often add an extra sidecar container that runs the probe script and exposes a Prometheus metric.

Example: Sidecar dependency probe for orders → inventory

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders
  namespace: orders
spec:
  template:
    spec:
      containers:
      - name: orders-app
        image: orders:1.4.0
        # ... main app ports, env, etc.
      - name: inventory-probe
        image: probehub/dependency-probe:latest
        env:
        - name: TARGET_HOST
          value: inventory.orders.svc.cluster.local
        - name: TARGET_PORT
          value: "8080"
        - name: TARGET_PATH
          value: "/inventory/health"
        - name: PROBE_INTERVAL
          value: "30s"
        - name: PROBE_TIMEOUT
          value: "5s"
        ports:
        - containerPort: 9100   # Prometheus metrics endpoint

The probe container runs a loop:

#!/usr/bin/env sh
while true; do
  start=$(date +%s%N)
  status=$(curl -s -o /dev/null -w "%{http_code}" \
    --cacert /etc/ssl/certs/ca-certificates.crt \
    https://"$TARGET_HOST:$TARGET_PORT$TARGET_PATH")
  latency=$(( ($(date +%s%N) - start) / 1000000 ))
  if [ "$status" -ge 200 ] && [ "$status" -lt 400 ]; then
    echo "probe_success{target=\"$TARGET_HOST\"} 1"
    echo "probe_latency_ms{target=\"$TARGET_HOST\"} $latency"
  else
    echo "probe_success{target=\"$TARGET_HOST\"} 0"
  fi
  sleep "$PROBE_INTERVAL"
done

Prometheus scrapes :9100/metrics and alerts if probe_success drops to 0 for more than two consecutive intervals.

Example CLI Commands for Probe Configuration

Istio sidecar injection with probe annotation

kubectl annotate deployment orders \
  sidecar.istio.io/inject=true \
  sidecar.istio.io/probe-container=inventory-probe \
  -n orders

Calico felixconfiguration to enable probe logs

calicoctl create -f - <<EOF
apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
  name: default
spec:
  logSeverityScreen: Info
  enableLogDrop: true   # logs packets dropped by policy, useful for probe failure analysis
EOF

OpenTelemetry collector to export probe traces

receivers:
  otlp:
    protocols:
      grpc:
processors:
  batch:
exporters:
  prometheus:
    endpoint: "0.0.0.0:9464"
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [prometheus]

Rolling Out Service-to-Service Policy Changes

Pre‑Deployment Checklist