Designing a CI Workflow for Identity-Based Segmentation When Labels, Tags, and Service Ownership Change Daily

Engineering design review memo – Sophia Lin, Cloud & Automation Architect

Introduction

Identity‑Based Segmentation

Identity‑based segmentation (IBS) enforces network policies using immutable or semi‑immutable workload attributes—labels, tags, or service‑ownership metadata—rather than static IP addresses. In Kubernetes or service‑mesh environments, a policy engine (e.g., Open Policy Agent, Istio AuthorizationPolicy) evaluates in‑flight requests against a rule set that references these attributes. When a workload’s label set changes, the applicable policy must be updated atomically to avoid unintended exposure or accidental denial‑of‑service.

Why CI Matters in Dynamic Environments

When labels, tags, and ownership are updated multiple times per day (feature‑branch deployments, autoscaling groups, dev self‑service namespaces), a manual policy‑update process becomes a bottleneck and a source of drift. CI provides a repeatable, automated gate that:

1. Validates proposed label/tag changes against the current segmentation model.
2. Generates the corresponding policy artefacts (OPA rules, Istio objects, firewall ACLs).
3. Enforces approval workflows before artefacts are promoted to the production policy store.

Without CI, operators risk pushing stale or contradictory policies, leading to overly permissive traffic (security incident) or overly restrictive traffic (service outage).

Designing the CI Workflow

Trust Boundaries

The trust boundary is enforced at the sidecar proxy (Envoy) via Istio AuthorizationPolicy, which consumes OPA‑generated JSON policy bundles.

• Policy Decision Point (PDP) – OPA evaluates incoming attributes against the rule set.
• Policy Enforcement Point (PEP) – Envoy sidecar applies the decision.

Assumption: the service mesh fetches policy bundles from a well‑known HTTP endpoint (e.g., http://policy-store:8080/bundle.tar.gz).

Placing the PDP inside the application container is rejected because it expands the attack surface and complicates artefact versioning.

Managing Labels, Tags, and Service Ownership

The source of truth for labels/tags/ownership lives in a GitOps repository (infra/label-store). Each file under labels/ represents a Kubernetes namespace or workload selector and contains a YAML map, e.g.:

# labels/frontend.yml
app: frontend
team: product-a
environment: prod
version: "1.4.2"

A CI pipeline watches this directory for changes and:

1. Reads the changed files.
2. Validates the schema (using jsonschema or a Pydantic model).
3. Generates OPA rule files (*.rego) that encode the allow‑list for each label combination.
4. Packages the rules into a versioned bundle (bundle.tar.gz) and pushes it to an immutable object store (versioned S3 or a dedicated policy registry).

Rejected alternatives

• Side‑car label injector (mutating webhook): does not affect existing pods and adds pod‑start latency.
• CMDB‑driven sync: introduces an external dependency and hinders isolated testing.

Approval Gates

Approval gates are separate stages in the GitLab CI pipeline:

The minimum safe approval gate for any change that touches the trust boundary is unit-test + review. The security-scan stage is strongly recommended but may be omitted in low‑risk environments with documented risk acceptance.

Managing Daily Changes

Automated Detection of Changes

GitLab’s trigger pipelines on push feature runs the pipeline only when files under labels/ change:

stages:
  - lint
  - unit-test
  - security-scan
  - review
  - promote

lint:
  stage: lint
  script:
    - yamllint labels/
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        - labels/**/*

If a GitOps approach is infeasible, a lightweight polling agent can query a label API (e.g., Consul KV) every 5 minutes, commit any diff to a temporary branch, and trigger the same pipeline.

Keeping the Pipeline Static

The pipeline itself remains static; only the generated artefact (the OPA bundle) changes. To avoid rewriting the CI config on each label change:

• A Makefile or bash script (generate_policy.sh) is invoked from the unit-test stage.
• The script reads the current state of labels/, runs a Jinja2 template to produce *.rego files, and executes opa build -b . -o bundle.tar.gz.

Thus the CI configuration does not need version bumps for each label update; only the bundle changes.

Mitigating Stale‑Cache Risks

Stale cache can appear in two places:

1. Envoy sidecar cache – Envoy caches the fetched policy bundle (TTL default 60 s).
2. GitLab runner artefact cache – runners may cache bundle.tar.gz between jobs if not disabled.

Envoy mitigations

• Set max_delay and max_retries in the typed_per_filter_config for the envoy.filters.http.rbac filter.
• Lower fetch_api_timeout (e.g., to 5 s).
• Enable xDS with resource_api_version: V3 and set max_stale: 0 to force a reload when the version hash changes.

CI pipeline mitigations

• Disable artefact caching for the bundle (artifacts: untracked: false) or explicitly delete the cache after promote:

promote:
  stage: promote
  script:
    - aws s3 cp bundle.tar.gz s3://policy-store/bundle-${CI_COMMIT_SHA}.tar.gz
    - curl -X POST http://mesh-controller/reload-policy
  when: manual
  only:
    - branches
  cache: false

If the bundle grows beyond a few megabytes, fetch latency may become noticeable—a scaling consideration for later tuning.

Troubleshooting

Stale‑Cache Errors

Symptom: Policy decisions reflect an old label set after a merge.

Diagnosis

1. Check Envoy sidecar logs for RBAC: cached policy version X.
2. Verify the SHA256 of bundle.tar.gz served by the policy store matches the latest commit (curl -I … returns an ETag).
3. Look for xDS: stale resource warnings if using xDS.

Resolution

• Force a sidecar restart: kubectl rollout restart deployment/<app>.
• Reduce Envoy policy fetch TTL via a MeshPolicy (configDiscoveryRefreshDelay: 1s).
• Ensure the promote stage includes a cache‑busting query parameter (?v=${CI_COMMIT_SHA}) when the mesh fetches the bundle.

Approval Gate Failures

Symptom: Pipeline stalls at the review stage with no approvers listed.

Diagnosis

• Inspect the merge request widget for missing approval rules (Approval Rules in GitLab).
• Confirm the MR author has sufficient permissions (Maintainer or higher).
• Review GitLab audit events for approval_rule_triggered failures.

Resolution

• Add the required approval rule (e.g., require approval from platform-engineers and security groups).
• If using CODEOWNERS, ensure changed labels/ files are owned by the appropriate group.
• As a last resort, manually retry the review stage via the GitLab UI (Retry button).

Trust Boundary Violations

Symptom: OPA logs show decision_id: DENY for a legitimate call, or ALLOW for a call that should be blocked.

Diagnosis

• Enable OPA decision logs: opa run --set=decision_logs.console=true.
• Correlate request attributes (source labels, destination labels, HTTP path) with the policy bundle version in use.
• Verify the label set used by the sidecar matches the source of truth (kubectl get pod -o jsonpath='{.metadata.labels}').

Resolution

• False positive (legitimate traffic denied): update label definitions in labels/ and re‑run the pipeline.
• False negative (undesired traffic allowed): inspect generated Rego rules for overly permissive wildcards (input.source.app == "*"), tighten the rule, and push a new bundle.
• Record the incident in the change‑management ticket and add a regression test to the unit-test stage to prevent recurrence.

Code and CLI Examples

GitLab CI Configuration

# .gitlab-ci.yml
stages:
  - lint
  - unit-test
  - security-scan
  - review
  - promote

variables:
  POLICY_STORE_BUCKET: policy-store
  POLICY_BUNDLE_PATH: bundle.tar.gz

lint:
  stage: lint
  script:
    - yamllint labels/
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      changes:
        - labels/**/*

unit-test:
  stage: unit-test
  script:
    - ./generate_policy.sh   # produces *.rego and bundle.tar.gz
    - opa test . -v
  artifacts:
    paths:
      - bundle.tar.gz
    expire_in: 1 hour

security-scan:
  stage: security-scan
  script:
    - conftest test -p policy bundle.tar.gz
  needs: ["unit-test"]

review:
  stage: review
  script:
    - echo "Waiting for manual approvals..."
  when: manual
  allow_failure: false

promote:
  stage: promote
  script:
    - aws s3 cp bundle.tar.gz s3://$POLICY_STORE_BUCKET/bundle-${CI_COMMIT_SHA}.tar.gz
    - curl -X POST http://mesh-controller/reload-policy?bundle=s3://$POLICY_STORE_BUCKET/bundle-${CI_COMMIT_SHA}.tar.gz
  when: manual
  only:
    - branches
  cache: false

Policy Generation Script (generate_policy.sh)

#!/usr/bin/env bash
set -euo pipefail

# Read label files and produce Rego rules
mkdir -p generated
for file in labels/*.yml; do
  base=$(basename "$file" .yml)
  yq eval '. as $item | "package {{ .app }}\nallow if {\n    input.source.app == \"{{ $item.app }}\"\n    input.source.team == \"{{ $item.team }}\"\n    input.destination.app == \"{{ $item.app }}\"\n}"' "$file" > "generated/${base}.rego"
done

# Build OPA bundle
opa build -b generated -o bundle.tar.gz

Make the script executable: chmod +x generate_policy.sh.

End of document.