Inspecting Cilium Network Policies: A Comprehensive Guide for EKS and Kubernetes

READER BEWARE: THE FOLLOWING WRITTEN ENTIRELY BY AI WITHOUT HUMAN EDITING.

Introduction

Cilium has emerged as a powerful networking solution for Kubernetes, offering advanced networking, observability, and security features through eBPF (extended Berkeley Packet Filter). One of Cilium’s core capabilities is its implementation of Network Policies, which provide fine-grained control over pod-to-pod communication, ingress/egress traffic, and service-level access control.

However, as your Kubernetes environment grows and network policies become more complex, understanding how to effectively inspect and troubleshoot these policies becomes critical. This guide provides a comprehensive overview of the various methods to inspect Cilium Network Policies, with special emphasis on Amazon EKS (Elastic Kubernetes Service) environments.

Understanding Cilium Network Policies

Before diving into inspection methods, let’s understand what Cilium Network Policies are and how they differ from standard Kubernetes Network Policies.

Cilium Network Policy vs Kubernetes Network Policy

Cilium supports two types of network policies:

Kubernetes NetworkPolicy: Standard Kubernetes API resources that Cilium can enforce
CiliumNetworkPolicy: Extended policy format with additional capabilities

Feature	Kubernetes NetworkPolicy	CiliumNetworkPolicy
API Group	networking.k8s.io/v1	cilium.io/v2
L7 Filtering	❌	✅ (HTTP, Kafka, gRPC, DNS)
Service-based Rules	❌	✅
FQDN-based Rules	❌	✅
Identity-based Rules	❌	✅
Visibility Annotations	❌	✅
Deny Policies	❌	✅ (CiliumClusterwideNetworkPolicy)

Cilium Policy Enforcement Model

Cilium uses a distributed security model where policies are enforced at each node’s eBPF datapath. Understanding this architecture helps explain why certain inspection methods are more effective than others:

┌─────────────────────────────────────────────────────────────┐
│                    Kubernetes API Server                     │
│         (NetworkPolicy, CiliumNetworkPolicy CRDs)            │
└────────────────────────┬────────────────────────────────────┘
                         │
                         ├─── Policy Distribution
                         │
        ┌────────────────┴────────────────┐
        │       Cilium Operator            │
        │   (Policy Compilation & Sync)    │
        └────────────────┬────────────────┘
                         │
        ┌────────────────┴────────────────┐
        │                                  │
┌───────▼────────┐              ┌────────▼────────┐
│  Cilium Agent  │              │  Cilium Agent   │
│    (Node 1)    │              │    (Node 2)     │
│                │              │                 │
│  ┌──────────┐  │              │  ┌──────────┐   │
│  │ eBPF Map │  │              │  │ eBPF Map │   │
│  │ Policies │  │              │  │ Policies │   │
│  └──────────┘  │              │  └──────────┘   │
└────────────────┘              └─────────────────┘

Prerequisites for Inspecting Cilium Policies

Before you begin inspecting Cilium Network Policies, ensure you have the appropriate tools and access:

Required Tools

kubectl: Kubernetes command-line tool
```
kubectl version --client
```

Cilium CLI: The official Cilium command-line interface

# Install Cilium CLI on macOS
brew install cilium-cli

# Install on Linux
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/stable/stable.txt)
CLI_ARCH=amd64
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

# Verify installation
cilium version

AWS CLI (for EKS-specific operations)
```
aws --version
```

jq: JSON processor for parsing outputs

# macOS
brew install jq

# Linux
sudo apt-get install jq

Required Permissions

For comprehensive policy inspection, you’ll need:

Kubernetes RBAC Permissions:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: cilium-policy-viewer
rules:
# View NetworkPolicies
- apiGroups: ["networking.k8s.io"]
  resources: ["networkpolicies"]
  verbs: ["get", "list", "watch"]
  
# View CiliumNetworkPolicies
- apiGroups: ["cilium.io"]
  resources: 
    - "ciliumnetworkpolicies"
    - "ciliumclusterwidenetworkpolicies"
    - "ciliumendpoints"
    - "ciliumidentities"
    - "ciliumnodes"
  verbs: ["get", "list", "watch"]
  
# Access to pods for exec (if needed for debugging)
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list"]
  
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]

AWS IAM Permissions (for EKS):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "eks:DescribeCluster",
        "eks:ListClusters",
        "eks:AccessKubernetesApi"
      ],
      "Resource": "*"
    }
  ]
}

Method 1: Inspecting Policies with kubectl

The most straightforward way to inspect Cilium Network Policies is through kubectl, which works with both standard Kubernetes NetworkPolicies and CiliumNetworkPolicies.

Listing Network Policies

List All Standard NetworkPolicies

# List NetworkPolicies in all namespaces
kubectl get networkpolicies --all-namespaces

# List with additional details
kubectl get networkpolicies -A -o wide

# Output example:
# NAMESPACE     NAME              POD-SELECTOR         AGE
# default       web-policy        app=web              5d
# production    api-policy        tier=api             3d

List All CiliumNetworkPolicies

# List CiliumNetworkPolicies in all namespaces
kubectl get ciliumnetworkpolicies --all-namespaces

# Short form
kubectl get cnp -A

# List CiliumClusterwideNetworkPolicies
kubectl get ciliumclusterwidenetworkpolicies

# Short form
kubectl get ccnp

Detailed Policy Inspection

View Policy Details in YAML Format

# Get full details of a NetworkPolicy
kubectl get networkpolicy web-policy -n default -o yaml

# Get full details of a CiliumNetworkPolicy
kubectl get cnp api-policy -n production -o yaml

# Get policy with custom columns
kubectl get cnp -A -o custom-columns=\
NAME:.metadata.name,\
NAMESPACE:.metadata.namespace,\
ENDPOINT-SELECTOR:.spec.endpointSelector

Example Output Analysis

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-backend-policy
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: api
      tier: backend
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - method: "GET|POST"
          path: "/api/.*"
  egress:
  - toEndpoints:
    - matchLabels:
        app: database
    toPorts:
    - ports:
      - port: "5432"
        protocol: TCP
  - toFQDNs:
    - matchName: "api.external-service.com"
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP

Using kubectl to Filter and Search Policies

Find Policies Affecting Specific Pods

# Get pod labels
kubectl get pod my-pod -n default --show-labels

# Find NetworkPolicies matching those labels
kubectl get networkpolicies -n default -o json | \
  jq '.items[] | select(.spec.podSelector.matchLabels.app=="web")'

# Find CiliumNetworkPolicies matching labels
kubectl get cnp -n default -o json | \
  jq '.items[] | select(.spec.endpointSelector.matchLabels.app=="web")'

Search for Policies with Specific Rules

# Find policies allowing traffic from a specific namespace
kubectl get cnp -A -o json | \
  jq '.items[] | select(.spec.ingress[]?.fromEndpoints[]?.matchLabels."k8s:io.kubernetes.pod.namespace"=="production")'

# Find policies with L7 HTTP rules
kubectl get cnp -A -o json | \
  jq '.items[] | select(.spec.ingress[]?.toPorts[]?.rules.http != null)'

# Find policies with FQDN rules
kubectl get cnp -A -o json | \
  jq '.items[] | select(.spec.egress[]?.toFQDNs != null)'

Analyzing Policy Coverage

# Count policies per namespace
kubectl get cnp -A -o json | \
  jq '.items | group_by(.metadata.namespace) | map({namespace: .[0].metadata.namespace, count: length})'

# List pods without network policies (in a namespace)
NAMESPACE="production"
kubectl get pods -n $NAMESPACE -o json | \
  jq -r '.items[] | .metadata.name' | while read pod; do
    LABELS=$(kubectl get pod $pod -n $NAMESPACE -o json | jq -r '.metadata.labels')
    POLICIES=$(kubectl get cnp -n $NAMESPACE -o json | \
      jq --argjson labels "$LABELS" '.items[] | select(.spec.endpointSelector.matchLabels as $selector | $labels | to_entries | map(select(.key as $k | .value as $v | $selector[$k] == $v)) | length > 0)')
    if [ -z "$POLICIES" ]; then
      echo "Pod $pod has no matching policies"
    fi
  done

Method 2: Using the Cilium CLI

The Cilium CLI provides powerful, purpose-built commands for inspecting and troubleshooting network policies.

Checking Cilium Status

# Check overall Cilium status
cilium status

# Check Cilium status with detailed output
cilium status --wait

# Output includes:
# - Cilium operator status
# - Number of endpoints managed
# - Policy enforcement status
# - Hubble status (if enabled)

Connectivity Testing

# Run connectivity test between pods
cilium connectivity test

# This creates test pods and validates:
# - Pod-to-pod connectivity
# - Pod-to-service connectivity
# - Policy enforcement
# - DNS resolution
# - External connectivity

Inspecting Cilium Endpoints

Cilium endpoints represent individual pods with their security identities and applied policies.

# List all Cilium endpoints
kubectl get ciliumendpoints -A

# Get detailed endpoint information
kubectl get ciliumendpoint <pod-name> -n <namespace> -o yaml

# Short form
kubectl get cep -A

Understanding Endpoint Output

apiVersion: cilium.io/v2
kind: CiliumEndpoint
metadata:
  name: api-pod-abc123
  namespace: production
status:
  external-identifiers:
    container-id: docker://abc123...
    pod-name: production/api-pod-abc123
  id: 1234
  identity:
    id: 5678
    labels:
    - k8s:app=api
    - k8s:tier=backend
    - k8s:io.kubernetes.pod.namespace=production
  networking:
    addressing:
    - ipv4: 10.0.1.23
  policy:
    realized:
      allowed-egress-identities:
      - 9012  # database identity
      - 1     # world identity (for external access)
      allowed-ingress-identities:
      - 3456  # frontend identity
      policy-enabled: both
      policy-revision: 42
    spec:
      policy-enabled: both
  state: ready

Key fields to examine:

identity.id: Security identity assigned to this endpoint
policy.realized.policy-enabled: Whether ingress/egress policies are enforced
policy.realized.allowed-ingress-identities: Which identities can send traffic to this endpoint
policy.realized.allowed-egress-identities: Which identities this endpoint can send traffic to
policy.realized.policy-revision: Version of the policy applied

Inspecting Cilium Identities

Cilium assigns security identities to pods based on their labels. Understanding identities is crucial for policy troubleshooting.

# List all Cilium identities
kubectl get ciliumidentities

# Short form
kubectl get ciliumid

# Get specific identity details
kubectl get ciliumidentity <id> -o yaml

# Find which identity is assigned to a pod
kubectl get cep <pod-name> -n <namespace> -o jsonpath='{.status.identity.id}'

Finding Pods by Identity

# List all endpoints with a specific identity
IDENTITY_ID=5678
kubectl get cep -A -o json | \
  jq -r ".items[] | select(.status.identity.id==$IDENTITY_ID) | \"\(.metadata.namespace)/\(.metadata.name)\""

Policy Validation

# Validate policy syntax (dry-run)
kubectl apply --dry-run=client -f my-policy.yaml

# Validate policy syntax (server-side)
kubectl apply --dry-run=server -f my-policy.yaml

Method 3: EKS-Specific Inspection Methods

When running Cilium on Amazon EKS, there are additional methods to inspect policies leveraging AWS services and EKS-specific features.

Accessing EKS Cluster with AWS CLI

# Configure kubectl for EKS cluster
aws eks update-kubeconfig --name my-cluster --region us-west-2

# List EKS clusters
aws eks list-clusters --region us-west-2

# Get cluster information
aws eks describe-cluster --name my-cluster --region us-west-2

Inspecting EKS with Cilium Network Policy Add-on

If you’re using EKS with the managed Cilium add-on:

# List EKS add-ons
aws eks list-addons --cluster-name my-cluster --region us-west-2

# Describe Cilium add-on
aws eks describe-addon \
  --cluster-name my-cluster \
  --addon-name vpc-cni \
  --region us-west-2

Using AWS Systems Manager for Direct Node Access

For deeper inspection, you can access EKS nodes directly using AWS Systems Manager (SSM):

# List EC2 instances in your EKS cluster
aws ec2 describe-instances \
  --filters "Name=tag:eks:cluster-name,Values=my-cluster" \
  --query 'Reservations[].Instances[].[InstanceId,PrivateIpAddress,State.Name]' \
  --output table

# Start SSM session to a node
aws ssm start-session --target i-1234567890abcdef0

# Once connected, inspect eBPF maps
sudo cilium bpf policy list

# View policy statistics
sudo cilium bpf policy get --all

CloudWatch Logs Integration

Configure Cilium to send logs to CloudWatch for centralized monitoring:

# ConfigMap for Cilium on EKS with CloudWatch
apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
data:
  debug: "false"
  enable-policy: "default"
  enable-hubble: "true"
  hubble-export-file-path: "/var/log/cilium/hubble.log"
  # Additional monitoring settings

Then use CloudWatch Insights to query policy decisions:

# CloudWatch Insights query for policy denials
fields @timestamp, @message
| filter @message like /policy/
| filter @message like /denied/
| sort @timestamp desc
| limit 100

Method 4: Using Cilium Agent Debug Commands

The Cilium agent running on each node provides rich debugging capabilities through its API.

Port-Forward to Cilium Agent

# Find Cilium agent pod on a specific node
kubectl get pods -n kube-system -l k8s-app=cilium -o wide

# Port-forward to Cilium agent
kubectl port-forward -n kube-system <cilium-pod-name> 9876:9876

Using cilium-dbg (Cilium Debug CLI)

# Execute commands in Cilium pod
kubectl exec -n kube-system <cilium-pod-name> -- cilium-dbg status

# List policy rules
kubectl exec -n kube-system <cilium-pod-name> -- cilium-dbg policy get

# Show policy for specific endpoint
kubectl exec -n kube-system <cilium-pod-name> -- cilium-dbg endpoint list
kubectl exec -n kube-system <cilium-pod-name> -- cilium-dbg endpoint get <endpoint-id>

# Trace policy decision
kubectl exec -n kube-system <cilium-pod-name> -- cilium-dbg policy trace \
  --src-identity <source-identity> \
  --dst-identity <destination-identity>

Policy Trace Example

# Example: Trace connectivity from frontend (identity 1234) to backend (identity 5678)
kubectl exec -n kube-system cilium-abc123 -- cilium-dbg policy trace \
  --src-identity 1234 \
  --dst-identity 5678 \
  --dport 8080

# Output shows:
# - Whether traffic is allowed or denied
# - Which policy rules were evaluated
# - The policy revision applied
# - Detailed verdict explanation

Monitor Policy Decisions in Real-Time

# Monitor policy decisions for a specific endpoint
kubectl exec -n kube-system <cilium-pod-name> -- \
  cilium-dbg monitor --type policy-verdict

# Monitor with filters
kubectl exec -n kube-system <cilium-pod-name> -- \
  cilium-dbg monitor --type policy-verdict --related-to <endpoint-id>

Method 5: Hubble for Advanced Observability

Hubble is Cilium’s observability platform that provides deep visibility into network flows and policy decisions.

Installing Hubble

# Install Hubble CLI
HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/stable/stable.txt)
HUBBLE_ARCH=amd64
curl -L --fail --remote-name-all \
  https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}
sha256sum --check hubble-linux-${HUBBLE_ARCH}.tar.gz.sha256sum
sudo tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz{,.sha256sum}

# Enable Hubble in Cilium
cilium hubble enable

# Verify Hubble is running
cilium hubble ui

Port-Forward to Hubble Relay

# Port-forward to Hubble Relay
cilium hubble port-forward

Observing Policy Decisions with Hubble

# Watch all flows
hubble observe

# Watch flows for a specific namespace
hubble observe --namespace production

# Watch flows for specific pods
hubble observe --pod production/api-pod

# Filter by verdict (allowed or denied)
hubble observe --verdict DROPPED
hubble observe --verdict FORWARDED

# Filter by protocol
hubble observe --protocol TCP --port 8080
hubble observe --protocol HTTP

Policy-Specific Hubble Queries

# Show denied flows (policy violations)
hubble observe --verdict DROPPED --type policy-verdict

# Show allowed flows for a specific endpoint
hubble observe --from-pod production/api-pod --verdict FORWARDED

# Show HTTP L7 policy decisions
hubble observe --protocol http --type l7

# Export flows to JSON for analysis
hubble observe --verdict DROPPED -o json > denied-flows.json

Hubble UI for Visual Inspection

# Open Hubble UI
cilium hubble ui

# This opens a browser showing:
# - Service map with traffic flows
# - Real-time network activity
# - Policy enforcement visualization
# - Flow details and statistics

Method 6: Policy Impact Analysis

Before applying or modifying policies, it’s crucial to understand their potential impact.

Dry-Run Policy Testing

# Test policy application without actually applying it
kubectl apply --dry-run=server -f new-policy.yaml

# View what would be changed
kubectl diff -f new-policy.yaml

Analyzing Policy Effectiveness

# Check if a policy is being applied to endpoints
NAMESPACE="production"
POLICY_NAME="api-policy"

# Get endpoints that should match the policy
kubectl get cep -n $NAMESPACE -o json | \
  jq -r --arg policy "$POLICY_NAME" \
  '.items[] | select(.status.policy.spec."policy-enabled" != "none") | 
   "\(.metadata.name): Policy enabled for \(.status.policy.spec."policy-enabled")"'

Identifying Over-Permissive Policies

# Find policies allowing traffic from anywhere
kubectl get cnp -A -o json | \
  jq '.items[] | select(.spec.ingress[]?.fromEndpoints == null and 
                         .spec.ingress[]?.fromEntities == null) | 
     {name: .metadata.name, namespace: .metadata.namespace}'

# Find policies with world egress
kubectl get cnp -A -o json | \
  jq '.items[] | select(.spec.egress[]?.toEntities[]? == "world") | 
     {name: .metadata.name, namespace: .metadata.namespace, egress: .spec.egress}'

Method 7: Troubleshooting Common Policy Issues

Issue 1: Policy Not Being Applied

Symptoms: Pods aren’t behaving according to the defined policy.

Diagnosis:

# 1. Verify policy exists and is valid
kubectl get cnp <policy-name> -n <namespace>

# 2. Check policy selector matches pod labels
POD_LABELS=$(kubectl get pod <pod-name> -n <namespace> -o json | jq '.metadata.labels')
POLICY_SELECTOR=$(kubectl get cnp <policy-name> -n <namespace> -o json | jq '.spec.endpointSelector')

echo "Pod labels: $POD_LABELS"
echo "Policy selector: $POLICY_SELECTOR"

# 3. Check endpoint shows policy is applied
kubectl get cep <pod-name> -n <namespace> -o jsonpath='{.status.policy.realized.policy-enabled}'

# 4. Verify policy revision is up-to-date
kubectl get cep <pod-name> -n <namespace> -o jsonpath='{.status.policy.realized.policy-revision}'

Common Causes:

Label mismatch between policy selector and pod labels
Policy in different namespace than pods (unless using CiliumClusterwideNetworkPolicy)
Cilium agent not running or outdated

Issue 2: Unexpected Traffic Blocked

Symptoms: Traffic that should be allowed is being dropped.

Diagnosis:

# 1. Check for denied flows in Hubble
hubble observe --verdict DROPPED --from-pod <namespace>/<pod-name>

# 2. Trace the policy decision
SOURCE_POD="frontend-pod"
DEST_POD="backend-pod"
NAMESPACE="production"

SOURCE_ID=$(kubectl get cep $SOURCE_POD -n $NAMESPACE -o jsonpath='{.status.identity.id}')
DEST_ID=$(kubectl get cep $DEST_POD -n $NAMESPACE -o jsonpath='{.status.identity.id}')

kubectl exec -n kube-system cilium-xyz -- cilium-dbg policy trace \
  --src-identity $SOURCE_ID \
  --dst-identity $DEST_ID \
  --dport 8080

# 3. Review all policies affecting the destination
kubectl get cnp -n $NAMESPACE -o json | \
  jq --arg dest "$DEST_POD" '.items[] | 
      select(.spec.endpointSelector | . != null)'

Common Causes:

Missing egress rule in source policy
Missing ingress rule in destination policy
Port or protocol mismatch
L7 rules blocking specific HTTP methods or paths
Conflicting deny policy (CiliumClusterwideNetworkPolicy)

Issue 3: DNS Resolution Failures

Symptoms: Pods can’t resolve external FQDNs despite policy allowing it.

Diagnosis:

# 1. Check if DNS proxy is enabled
kubectl exec -n kube-system cilium-xyz -- cilium-dbg config | grep dns-proxy

# 2. Verify DNS policy rules
kubectl get cnp -A -o json | \
  jq '.items[] | select(.spec.egress[]?.toFQDNs != null)'

# 3. Check DNS visibility in Hubble
hubble observe --protocol DNS

# 4. Test DNS resolution from pod
kubectl exec <pod-name> -n <namespace> -- nslookup example.com

Common Causes:

DNS proxy not enabled in Cilium
Missing toPort 53 rule for DNS
FQDN policy not using correct matchName pattern
DNS caching issues

Issue 4: Policy Performance Impact

Symptoms: Increased latency after applying policies.

Diagnosis:

# 1. Check eBPF map sizes
kubectl exec -n kube-system cilium-xyz -- cilium-dbg bpf policy list

# 2. Monitor policy processing time
kubectl exec -n kube-system cilium-xyz -- cilium-dbg monitor --type policy-verdict | \
  grep -E "Processing time"

# 3. Check for policy complexity
kubectl get cnp -A -o json | \
  jq '.items[] | {
    name: .metadata.name,
    ingress_rules: (.spec.ingress | length),
    egress_rules: (.spec.egress | length),
    l7_rules: ([.spec.ingress[]?, .spec.egress[]? | 
                .toPorts[]?.rules.http?] | flatten | length)
  }'

Solutions:

Consolidate multiple policies where possible
Optimize L7 rules (they have more overhead than L3/L4)
Use identity-based rules instead of label selectors when possible

Method 8: Automating Policy Inspection

For continuous monitoring and compliance checking, automate policy inspection.

Shell Script for Policy Auditing

#!/bin/bash
# cilium-policy-audit.sh

NAMESPACES=$(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}')

echo "=== Cilium Network Policy Audit ==="
echo "Generated: $(date)"
echo ""

for NS in $NAMESPACES; do
  echo "Namespace: $NS"
  echo "----------------------------------------"
  
  # Count policies
  CNP_COUNT=$(kubectl get cnp -n $NS --no-headers 2>/dev/null | wc -l)
  NP_COUNT=$(kubectl get networkpolicies -n $NS --no-headers 2>/dev/null | wc -l)
  
  echo "  CiliumNetworkPolicies: $CNP_COUNT"
  echo "  NetworkPolicies: $NP_COUNT"
  
  # Check for unprotected pods
  PODS=$(kubectl get pods -n $NS -o json 2>/dev/null | jq -r '.items[].metadata.name')
  UNPROTECTED=0
  
  for POD in $PODS; do
    HAS_POLICY=$(kubectl get cep $POD -n $NS -o jsonpath='{.status.policy.realized.policy-enabled}' 2>/dev/null)
    if [ "$HAS_POLICY" = "none" ] || [ -z "$HAS_POLICY" ]; then
      ((UNPROTECTED++))
    fi
  done
  
  echo "  Unprotected Pods: $UNPROTECTED"
  echo ""
done

echo "=== Cluster-wide Policies ==="
kubectl get ccnp --no-headers | wc -l
echo ""

echo "=== Total Cilium Identities ==="
kubectl get ciliumidentities --no-headers | wc -l

Policy Validation CI/CD Integration

# .github/workflows/validate-policies.yml
name: Validate Cilium Policies

on:
  pull_request:
    paths:
      - 'k8s/policies/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup kubectl
        uses: azure/setup-kubectl@v3
      
      - name: Install Cilium CLI
        run: |
          CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
          curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
          sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin          
      
      - name: Validate Policy Syntax
        run: |
          for policy in k8s/policies/*.yaml; do
            echo "Validating $policy"
            kubectl apply --dry-run=client -f $policy
          done          
      
      - name: Check Policy Conflicts
        run: |
          # Custom script to detect overlapping policies
          ./scripts/check-policy-conflicts.sh

Method 9: Monitoring and Alerting

Set up continuous monitoring for policy health and violations.

Prometheus Metrics

Cilium exposes metrics that can be scraped by Prometheus:

# ServiceMonitor for Cilium
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cilium-agent
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: cilium
  endpoints:
  - port: metrics
    interval: 30s

Key Metrics to Monitor:

# Policy enforcement status
cilium_policy_enforcement_status

# Number of policy drops
rate(cilium_drop_count_total{reason="Policy denied"}[5m])

# Policy change events
rate(cilium_policy_change_total[5m])

# Endpoint policy revision
cilium_endpoint_policy_revision

# Identity changes
rate(cilium_identity_label_sources_total[5m])

Alerting Rules

# PrometheusRule for Cilium policy alerts
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cilium-policy-alerts
  namespace: kube-system
spec:
  groups:
  - name: cilium.policies
    interval: 30s
    rules:
    - alert: HighPolicyDenyRate
      expr: rate(cilium_drop_count_total{reason="Policy denied"}[5m]) > 10
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High rate of policy denials detected"
        description: "Cilium is dropping packets due to policy at {{ $value }} drops/sec"
    
    - alert: PolicyEnforcementDisabled
      expr: cilium_policy_enforcement_status == 0
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "Policy enforcement is disabled"
        description: "Cilium policy enforcement is disabled on {{ $labels.instance }}"
    
    - alert: EndpointPolicyOutOfSync
      expr: max(cilium_endpoint_policy_revision) by (endpoint) < max(cilium_policy_revision) by (endpoint)
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "Endpoint policy out of sync"
        description: "Endpoint {{ $labels.endpoint }} policy revision is behind"

Logging Best Practices

Configure structured logging for policy events:

# Cilium ConfigMap with enhanced logging
apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
data:
  debug: "false"
  debug-verbose: "flow datapath"  # Enable flow and datapath debugging
  enable-policy: "default"
  policy-audit-mode: "false"      # Set to true to log all policy decisions
  monitor-aggregation: "medium"

Method 10: Best Practices for Policy Management

Policy Organization

# Organize policies by environment and tier
# File: production/backend/api-policy.yaml
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-backend
  namespace: production
  labels:
    environment: production
    tier: backend
    team: platform
  annotations:
    description: "Restricts API backend to only accept traffic from frontend"
spec:
  endpointSelector:
    matchLabels:
      app: api
      tier: backend
  ingress:
  - fromEndpoints:
    - matchLabels:
        app: frontend
        tier: frontend
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP

Documentation Standards

Include clear documentation in policy annotations:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payment-service-policy
  namespace: production
  annotations:
    policy.cilium.io/description: |
      Payment service network policy
      - Allows ingress from API gateway only
      - Allows egress to payment processor and database
      - Blocks all other traffic      
    policy.cilium.io/owner: "payments-team"
    policy.cilium.io/last-reviewed: "2025-12-01"
    policy.cilium.io/compliance: "PCI-DSS"
spec:
  # ... policy rules ...

Testing Policies in Staging

Always test policies in a non-production environment first:

# 1. Apply policy in staging namespace
kubectl apply -f policy.yaml -n staging

# 2. Run connectivity tests
cilium connectivity test --namespace staging

# 3. Monitor for issues
hubble observe --namespace staging --verdict DROPPED

# 4. Validate with real traffic simulation
kubectl run test-pod --image=curlimages/curl -n staging -- \
  sleep 3600

kubectl exec -n staging test-pod -- curl http://api-service:8080

# 5. Check metrics
kubectl top pods -n staging

Conclusion

Inspecting Cilium Network Policies effectively requires familiarity with multiple tools and methods:

kubectl: Basic policy viewing and filtering
Cilium CLI: Purpose-built commands for Cilium-specific resources
EKS Integration: Leveraging AWS services for enhanced visibility
Agent Debug Commands: Deep dive into policy enforcement at the node level
Hubble: Real-time flow observation and policy decision tracking
Automation: Scripts and CI/CD integration for continuous validation
Monitoring: Prometheus metrics and alerting for proactive management

By combining these methods, you can maintain comprehensive visibility into your network policies, quickly troubleshoot issues, and ensure your security posture remains strong as your Kubernetes environment evolves.

Key Takeaways

Start with kubectl for basic inspection, then use specialized tools for deeper analysis
Always use Hubble for troubleshooting policy issues—it provides invaluable real-time visibility
In EKS environments, leverage AWS-native tools for additional context
Automate policy validation in CI/CD pipelines to catch issues early
Monitor policy metrics continuously to detect enforcement problems
Document policies thoroughly and test in staging before production deployment
Regularly audit policies to ensure they remain necessary and effective

Additional Resources

Troubleshooting Quick Reference

Symptom	Tool to Use	Command
Traffic unexpectedly blocked	Hubble	`hubble observe --verdict DROPPED`
Policy not applying	kubectl	`kubectl get cep <pod> -n <ns> -o yaml`
Slow policy enforcement	cilium-dbg	`cilium-dbg monitor --type policy-verdict`
DNS resolution failing	Hubble	`hubble observe --protocol DNS`
Identity confusion	kubectl	`kubectl get ciliumidentities`
Policy conflicts	cilium-dbg	`cilium-dbg policy trace`
EKS-specific issues	AWS CLI	`aws eks describe-cluster`
Performance degradation	Prometheus	Check `cilium_drop_count_total`