Container Image Signing with Cosign and Policy Enforcement with Kyverno on AWS EKS

READER BEWARE: THE FOLLOWING WRITTEN ENTIRELY BY AI WITHOUT HUMAN EDITING.

Introduction

In modern container-based software delivery pipelines, ensuring the integrity and authenticity of container images is paramount. As organizations move towards cloud-native architectures, the attack surface expands, making it critical to implement robust security controls. Two powerful open-source tools address this challenge: Cosign for signing and verifying container images, and Kyverno for enforcing policy-based security controls in Kubernetes clusters.

This comprehensive guide explores how to integrate Cosign and Kyverno in an AWS EKS (Elastic Kubernetes Service) cluster to create a secure, verifiable software supply chain. We’ll cover the fundamentals of both tools, implementation details, practical examples, and the security value they provide to your organization.

The Container Security Challenge

Before diving into the solution, let’s understand the problem:

Software Supply Chain Risks

Modern software delivery faces several critical security challenges:

1. Image Tampering: Container images can be modified between build and deployment
2. Unauthorized Images: Attackers can push malicious images to registries
3. Lack of Provenance: Difficult to verify where an image came from and who built it
4. Registry Compromise: If a registry is compromised, all downstream systems are at risk
5. Compliance Requirements: Regulations require proof of image integrity and authenticity

Traditional Approaches and Their Limitations

Traditional security measures often fall short:

• Registry Access Controls: Only control who can push/pull, not image integrity
• Vulnerability Scanning: Identifies known CVEs but doesn’t prevent tampering
• Manual Reviews: Don’t scale and are error-prone
• Image Digests: Provide immutability but not authenticity

What’s needed: A cryptographic signing mechanism combined with automated policy enforcement that operates at admission time in Kubernetes.

What is Cosign?

Cosign is a container signing and verification tool developed by Sigstore, a Linux Foundation project. It provides cryptographic signing, verification, and storage for container images and other artifacts.

Key Features of Cosign

1. Container Image Signing: Sign OCI (Open Container Initiative) images with private keys
2. Keyless Signing: Use OpenID Connect (OIDC) identity tokens for ephemeral signing
3. Transparency Log: All signatures are recorded in Rekor, a transparency log
4. Attestation Support: Store SBOM (Software Bill of Materials) and build provenance
5. Multiple Key Management Options: Hardware tokens, KMS (Key Management Service), or key files
6. Cosign Verification: Verify signatures before deployment

How Cosign Works

┌─────────────────────────────────────────────────────────────────┐
│                     Cosign Signing Flow                          │
├─────────────────────────────────────────────────────────────────┤
│                                                                   │
│  1. Build Container Image                                        │
│     └─> docker build -t myapp:v1.0                              │
│                                                                   │
│  2. Push to Registry                                             │
│     └─> docker push myregistry.io/myapp:v1.0                    │
│                                                                   │
│  3. Sign Image with Cosign                                       │
│     └─> cosign sign --key cosign.key myregistry.io/myapp:v1.0  │
│         • Generates cryptographic signature                      │
│         • Stores signature in registry as OCI artifact           │
│         • Records signature in Rekor transparency log            │
│                                                                   │
│  4. Verify Signature (at deployment)                             │
│     └─> cosign verify --key cosign.pub myregistry.io/myapp:v1.0│
│         • Validates signature against public key                 │
│         • Checks transparency log                                │
│         • Confirms image hasn't been tampered                    │
│                                                                   │
└─────────────────────────────────────────────────────────────────┘

Cosign Signing Methods

Cosign supports multiple signing methods:

1. Key-Based Signing (Traditional)

Uses a generated key pair:

# Generate key pair
cosign generate-key-pair

# Sign image
cosign sign --key cosign.key myregistry.io/myapp:v1.0

# Verify image
cosign verify --key cosign.pub myregistry.io/myapp:v1.0

Pros: Full control, works offline Cons: Key management complexity, rotation challenges

2. Keyless Signing (OIDC-Based)

Uses ephemeral keys tied to your identity:

# Sign with keyless mode (uses OIDC identity)
cosign sign myregistry.io/myapp:v1.0

# Verify with issuer and identity
cosign verify \
  --certificate-identity=user@company.com \
  --certificate-oidc-issuer=https://accounts.google.com \
  myregistry.io/myapp:v1.0

Pros: No key management, identity-based, transparency by default Cons: Requires internet access, depends on OIDC provider

3. KMS-Based Signing

Uses cloud provider key management:

# Sign with AWS KMS
cosign sign --key awskms:///arn:aws:kms:us-east-1:123456789012:key/abcd-1234 \
  myregistry.io/myapp:v1.0

# Sign with Google Cloud KMS
cosign sign --key gcpkms://projects/my-project/locations/global/keyRings/my-ring/cryptoKeys/my-key \
  myregistry.io/myapp:v1.0

Pros: Centralized key management, audit trails, HSM-backed Cons: Cloud provider dependency, potential costs

What is Kyverno?

Kyverno is a policy engine designed specifically for Kubernetes. Unlike general-purpose policy engines, Kyverno is Kubernetes-native, using familiar YAML manifests and Kubernetes resources for policy definitions.

Key Features of Kyverno

1. Policy as Code: Define policies in Kubernetes CustomResources
2. No New Language: Uses Kubernetes YAML (no Rego or other DSL)
3. Three Policy Types:
   • Validation: Accept or reject resources based on rules
   • Mutation: Modify resources as they’re created
   • Generation: Create new resources automatically
4. Image Verification: Built-in support for Cosign signature verification
5. Reporting: Generate policy reports for compliance auditing
6. Admission Control: Operates at admission webhook level

How Kyverno Works

┌─────────────────────────────────────────────────────────────────┐
│              Kyverno Admission Control Flow                      │
├─────────────────────────────────────────────────────────────────┤
│                                                                   │
│  1. User/CI Creates Pod                                          │
│     └─> kubectl apply -f deployment.yaml                         │
│              │                                                    │
│              ▼                                                    │
│  2. API Server Intercepts Request                                │
│     └─> Sends to Kyverno Webhook                                │
│              │                                                    │
│              ▼                                                    │
│  3. Kyverno Evaluates Policies                                   │
│     ├─> Check image signatures with Cosign                       │
│     ├─> Validate resource configurations                         │
│     └─> Apply mutations if needed                                │
│              │                                                    │
│              ▼                                                    │
│  4. Decision                                                      │
│     ├─> ALLOW: Pod is created                                    │
│     └─> DENY: Pod creation rejected with error                   │
│                                                                   │
└─────────────────────────────────────────────────────────────────┘

Kyverno vs Other Policy Engines

Setting Up Cosign for Container Image Signing

Let’s walk through setting up Cosign in a CI/CD pipeline for signing container images.

Installation

Installing Cosign CLI

# Linux/macOS (using homebrew)
brew install sigstore/tap/cosign

# Linux (binary download)
wget https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64
sudo mv cosign-linux-amd64 /usr/local/bin/cosign
sudo chmod +x /usr/local/bin/cosign

# Verify installation
cosign version

Generating Keys

For production use with Kyverno, we’ll use key-based signing:

# Generate a key pair (will prompt for password)
cosign generate-key-pair

# This creates two files:
# - cosign.key (private key - keep secure!)
# - cosign.pub (public key - distribute to verify)

Important: Store the private key (cosign.key) securely:

• Use a secrets manager (AWS Secrets Manager, HashiCorp Vault)
• Encrypt at rest
• Restrict access with IAM policies
• Enable audit logging
• Consider using AWS KMS instead for better security

Signing Container Images

In CI/CD Pipeline (GitHub Actions Example)

name: Build and Sign Container Image

on:
  push:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  build-and-sign:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write  # For keyless signing

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=sha
            type=ref,event=branch
            type=semver,pattern={{version}}            

      - name: Build and push image
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

      - name: Sign container image
        env:
          COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
          COSIGN_PRIVATE_KEY: ${{ secrets.COSIGN_PRIVATE_KEY }}
        run: |
          cosign sign --yes --key env://COSIGN_PRIVATE_KEY \
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}          

      - name: Verify signature
        env:
          COSIGN_PUBLIC_KEY: ${{ secrets.COSIGN_PUBLIC_KEY }}
        run: |
          cosign verify --key env://COSIGN_PUBLIC_KEY \
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}          

Using AWS KMS for Signing

For AWS-centric workflows, use KMS:

# Create KMS key for signing
aws kms create-key \
  --description "Cosign container signing key" \
  --key-usage SIGN_VERIFY \
  --customer-master-key-spec RSA_4096

# Note the KeyId from output

# Sign image with KMS
export KMS_KEY_ID="arn:aws:kms:us-east-1:123456789012:key/abcd-1234"
cosign sign --key awskms:///${KMS_KEY_ID} \
  123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp:v1.0

Storing Signatures

Cosign stores signatures as OCI artifacts in the same registry as the image:

# Original image
myregistry.io/myapp:v1.0

# Signature (automatically stored)
myregistry.io/myapp:sha256-[digest].sig

This approach means:

• No separate signature storage needed
• Signatures follow images during replication
• Registry access controls apply to signatures

Deploying Kyverno to AWS EKS Cluster

Now let’s deploy Kyverno to an EKS cluster and configure it to verify Cosign signatures.

Prerequisites

Ensure you have:

# AWS CLI configured
aws sts get-caller-identity

# kubectl configured for your EKS cluster
kubectl cluster-info

# Helm installed
helm version

Creating an EKS Cluster (Optional)

If you need to create an EKS cluster:

# Using eksctl
eksctl create cluster \
  --name cosign-kyverno-demo \
  --region us-east-1 \
  --nodegroup-name standard-workers \
  --node-type t3.medium \
  --nodes 3 \
  --nodes-min 2 \
  --nodes-max 4 \
  --managed

# Update kubeconfig
aws eks update-kubeconfig --name cosign-kyverno-demo --region us-east-1

Installing Kyverno

Method 1: Using Helm (Recommended)

# Add Kyverno Helm repository
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

# Install Kyverno in its own namespace
helm install kyverno kyverno/kyverno \
  --namespace kyverno \
  --create-namespace \
  --set replicaCount=3 \
  --set image.pullPolicy=IfNotPresent

# Verify installation
kubectl get pods -n kyverno
kubectl get validatingwebhookconfigurations | grep kyverno
kubectl get mutatingwebhookconfigurations | grep kyverno

Method 2: Using kubectl

# Install latest stable release
kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.11.0/install.yaml

# Verify
kubectl get pods -n kyverno

Configuring Kyverno for High Availability on EKS

For production deployments on EKS:

# kyverno-values.yaml
replicaCount: 3

# Resource limits
resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 250m
    memory: 256Mi

# Pod anti-affinity to spread across nodes
affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/name
            operator: In
            values:
            - kyverno
        topologyKey: kubernetes.io/hostname

# Enable service monitor for Prometheus
metricsService:
  create: true
  type: ClusterIP
  port: 8000
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8000"

# Configure webhook timeout
webhooksTimeoutSeconds: 10

# Enable background scanning
backgroundController:
  enabled: true
  replicas: 2

Install with custom values:

helm install kyverno kyverno/kyverno \
  --namespace kyverno \
  --create-namespace \
  -f kyverno-values.yaml

Verifying Kyverno Installation

# Check Kyverno pods are running
kubectl get pods -n kyverno

# Expected output:
# NAME                      READY   STATUS    RESTARTS   AGE
# kyverno-5d5b5b5b5-xxxxx   1/1     Running   0          2m
# kyverno-5d5b5b5b5-yyyyy   1/1     Running   0          2m
# kyverno-5d5b5b5b5-zzzzz   1/1     Running   0          2m

# Check webhook configurations
kubectl get validatingwebhookconfigurations | grep kyverno
kubectl get mutatingwebhookconfigurations | grep kyverno

# View Kyverno logs
kubectl logs -n kyverno -l app.kubernetes.io/name=kyverno --tail=50

Integrating Cosign with Kyverno

Now comes the powerful integration: configuring Kyverno to automatically verify Cosign signatures when pods are deployed.

Step 1: Store Cosign Public Key in Kubernetes

First, create a ConfigMap or Secret containing the Cosign public key:

# Create secret with public key
kubectl create secret generic cosign-public-key \
  --from-file=cosign.pub=./cosign.pub \
  --namespace kyverno

# Or create a ConfigMap
kubectl create configmap cosign-public-key \
  --from-file=cosign.pub=./cosign.pub \
  --namespace kyverno

Step 2: Create Kyverno ClusterPolicy for Image Verification

Create a ClusterPolicy that requires all images to be signed:

# verify-images-policy.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
  annotations:
    policies.kyverno.io/title: Verify Container Image Signatures
    policies.kyverno.io/category: Security
    policies.kyverno.io/severity: high
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      Verifies that all container images are signed with Cosign.
      Images without valid signatures will be rejected.      
spec:
  validationFailureAction: Enforce
  background: false
  webhookTimeoutSeconds: 30
  failurePolicy: Fail
  rules:
    - name: verify-cosign-signature
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
      - imageReferences:
        - "*"
        attestors:
        - count: 1
          entries:
          - keys:
              publicKeys: |-
                -----BEGIN PUBLIC KEY-----
                MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEZh0kSYqvN0YNrKB6p8HKyKHKqGZW
                [... your actual public key content ...]
                -----END PUBLIC KEY-----                

Apply the policy:

kubectl apply -f verify-images-policy.yaml

Step 3: Advanced Policy Configurations

Policy with Multiple Registries

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-multiple-registries
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-ghcr-images
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
      - imageReferences:
        - "ghcr.io/*"
        attestors:
        - count: 1
          entries:
          - keys:
              publicKeys: |-
                -----BEGIN PUBLIC KEY-----
                [GHCR public key]
                -----END PUBLIC KEY-----                
    
    - name: verify-ecr-images
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
      - imageReferences:
        - "*.dkr.ecr.*.amazonaws.com/*"
        attestors:
        - count: 1
          entries:
          - keys:
              publicKeys: |-
                -----BEGIN PUBLIC KEY-----
                [ECR public key]
                -----END PUBLIC KEY-----                

Policy with Namespace Exemptions

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-images-with-exemptions
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-signed-images
      match:
        any:
        - resources:
            kinds:
              - Pod
      exclude:
        any:
        - resources:
            namespaces:
              - kube-system
              - kyverno
              - monitoring
      verifyImages:
      - imageReferences:
        - "*"
        attestors:
        - count: 1
          entries:
          - keys:
              publicKeys: |-
                -----BEGIN PUBLIC KEY-----
                [Your public key]
                -----END PUBLIC KEY-----                

Policy Using Secret Reference

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-images-with-secret
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-signature
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
      - imageReferences:
        - "*"
        attestors:
        - count: 1
          entries:
          - keys:
              secret:
                name: cosign-public-key
                namespace: kyverno

Step 4: Testing the Integration

Test 1: Deploy Signed Image

# Sign an image
cosign sign --key cosign.key myregistry.io/myapp:v1.0

# Create deployment with signed image
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: signed-app
spec:
  containers:
  - name: app
    image: myregistry.io/myapp:v1.0
EOF

# Expected: Pod created successfully

Test 2: Deploy Unsigned Image

# Try to deploy unsigned image
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: unsigned-app
spec:
  containers:
  - name: app
    image: myregistry.io/untrusted:latest
EOF

# Expected: Error message from Kyverno
# Error: admission webhook "mutate.kyverno.svc" denied the request:
# policy verify-image-signatures/verify-cosign-signature failed:
# failed to verify image signature: no matching signatures found

Step 5: Monitoring and Reporting

View policy reports:

# List policy reports
kubectl get policyreports -A

# View specific report
kubectl get policyreport -n default -o yaml

# Check Kyverno events
kubectl get events -n kyverno

# View detailed policy report
kubectl describe clusterpolicyreport cluster-policy-report

Using AWS KMS with Kyverno

For AWS-native deployments, integrate AWS KMS:

Step 1: Create KMS Key

# Create KMS key
aws kms create-key \
  --description "Cosign verification key" \
  --key-usage SIGN_VERIFY \
  --customer-master-key-spec RSA_4096

# Create alias
aws kms create-alias \
  --alias-name alias/cosign-verification \
  --target-key-id <key-id-from-previous-command>

Step 2: Configure IAM for EKS Nodes

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kms:GetPublicKey",
        "kms:Verify"
      ],
      "Resource": "arn:aws:kms:us-east-1:123456789012:key/*"
    }
  ]
}

Attach this policy to your EKS node IAM role.

Step 3: Configure Kyverno to Use KMS

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-with-kms
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-kms-signature
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
      - imageReferences:
        - "*.dkr.ecr.*.amazonaws.com/*"
        attestors:
        - count: 1
          entries:
          - keys:
              kms: "awskms:///arn:aws:kms:us-east-1:123456789012:key/abcd-1234"

Security Benefits and Value Proposition

Security Improvements

1. Supply Chain Security

Before Cosign + Kyverno:

• No way to verify image origin
• Images could be tampered between build and deployment
• Difficult to audit what’s running in production

After Cosign + Kyverno:

• Cryptographic proof of image authenticity
• Tampering detected automatically
• Complete audit trail via transparency logs
• Only signed, approved images can run

2. Reduced Attack Surface

Traditional Setup:
┌──────────────────────────────────────────────────┐
│  Threat Actor can:                                │
│  ✗ Push malicious image to registry              │
│  ✗ Modify image in transit                       │
│  ✗ Replace legitimate image                      │
│  ✗ Deploy unapproved versions                    │
└──────────────────────────────────────────────────┘

With Cosign + Kyverno:
┌──────────────────────────────────────────────────┐
│  Threat Actor blocked because:                    │
│  ✓ Cannot sign images without private key        │
│  ✓ Modified images fail verification             │
│  ✓ Unsigned images rejected at admission         │
│  ✓ All signatures logged in transparency log     │
└──────────────────────────────────────────────────┘

3. Compliance and Audit

Meets requirements for:

• SLSA (Supply-chain Levels for Software Artifacts): Level 2-3 compliance
• SOC 2: Demonstrable controls over software deployment
• ISO 27001: Cryptographic controls for software integrity
• NIST 800-53: Supply chain risk management (SR family)
• Executive Order 14028: Federal software supply chain requirements

Business Value

1. Cost Savings

• Incident Prevention: Avoid costs of security breaches ($4.45M average per breach - IBM 2023)
• Reduced Manual Reviews: Automated verification saves engineering time
• Faster Audits: Machine-readable signatures expedite compliance audits
• Insurance Benefits: May reduce cyber insurance premiums

2. Developer Productivity

• Clear Process: Developers know exactly what’s required
• Fast Feedback: Immediate rejection of unsigned images
• Self-Service: No waiting for security team approvals
• CI/CD Integration: Seamless integration into existing pipelines

3. Risk Reduction

Risk Matrix:

Use Case Examples

Use Case 1: Financial Services

Scenario: Bank deploying microservices to EKS Requirements:

• PCI DSS compliance
• Audit trail for all deployments
• No unauthorized software

Solution:

• All images signed with KMS keys
• Kyverno enforces signature verification
• Rekor provides immutable audit log
• Regular compliance reports generated

Outcome:

• Passed PCI DSS audit first try
• Zero unauthorized deployments
• Reduced audit preparation time by 60%

Use Case 2: Healthcare SaaS

Scenario: Healthcare platform on EKS Requirements:

• HIPAA compliance
• Software provenance tracking
• Change management controls

Solution:

• Images signed by CI/CD after security scans
• Kyverno verifies signatures + attestations
• SBOM attached to each image
• Automated policy reports

Outcome:

• HIPAA compliance maintained
• Complete software inventory
• Faster mean time to deployment

Use Case 3: Multi-Tenant Platform

Scenario: SaaS provider with customer workloads Requirements:

• Prevent cross-tenant image use
• Ensure only approved images run
• Rapid detection of anomalies

Solution:

• Namespace-specific signing keys
• Kyverno policies per namespace
• Different attestation requirements
• Real-time policy violations alerts

Outcome:

• Zero cross-tenant image deployments
• Customer confidence increased
• Faster security incident response

Best Practices

1. Key Management

DO:

• Use AWS KMS or cloud KMS when possible
• Rotate keys regularly (quarterly recommended)
• Store private keys in secrets manager
• Use separate keys for different environments
• Enable key audit logging

DON’T:

• Store keys in Git repositories
• Share keys across teams
• Use same key for dev and prod
• Hardcode keys in CI/CD pipelines

2. Policy Configuration

DO:

• Start with Audit mode before Enforce
• Use namespace exemptions carefully
• Set appropriate webhook timeouts
• Monitor policy report metrics
• Document policy decisions

DON’T:

• Apply policies to kube-system initially
• Set overly aggressive timeouts
• Exempt too many namespaces
• Ignore policy violation alerts

3. CI/CD Integration

DO:

• Sign immediately after building
• Verify signature before pushing
• Include signing in deployment gates
• Fail fast on signature errors
• Log all signing operations

DON’T:

• Skip signature verification
• Use unsigned images in prod
• Bypass signing for “urgent” changes
• Store keys in CI/CD logs

4. Monitoring and Alerting

Set up alerts for:

• Failed signature verifications
• Policy violations
• Unsigned image attempts
• Key usage anomalies
• Webhook timeouts

# Example Prometheus alert
- alert: UnsignedImageBlocked
  expr: kyverno_policy_results_total{policy="verify-image-signatures",result="fail"} > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Unsigned images blocked by Kyverno"
    description: "{{ $value }} unsigned images blocked in last 5 minutes"

Troubleshooting Common Issues

Issue 1: Image Verification Fails

Symptoms:

Error: admission webhook denied the request:
policy verify-image-signatures failed: failed to verify image

Diagnosis:

# Check if image is signed
cosign verify --key cosign.pub myregistry.io/myapp:v1.0

# Check Kyverno logs
kubectl logs -n kyverno -l app.kubernetes.io/name=kyverno | grep -i verify

# Validate public key in policy
kubectl get clusterpolicy verify-image-signatures -o yaml

Solutions:

1. Ensure image is actually signed
2. Verify public key matches private key used for signing
3. Check image reference format (digest vs tag)
4. Ensure registry credentials are available

Issue 2: Webhook Timeout

Symptoms:

Error: context deadline exceeded

Diagnosis:

# Check webhook timeout setting
kubectl get clusterpolicy -o yaml | grep webhookTimeoutSeconds

# Check Kyverno pod resource usage
kubectl top pods -n kyverno

Solutions:

# Increase webhook timeout
spec:
  webhookTimeoutSeconds: 30

# Increase Kyverno resources
resources:
  limits:
    cpu: 1000m
    memory: 1Gi

Issue 3: Registry Authentication

Symptoms:

Error: failed to resolve image reference: authentication required

Diagnosis:

# Check if imagePullSecrets exist
kubectl get secrets -n default

# Verify Kyverno can pull images
kubectl logs -n kyverno -l app.kubernetes.io/name=kyverno | grep -i auth

Solutions:

# Add imagePullSecrets to policy
spec:
  rules:
    - name: verify-signature
      verifyImages:
      - imageReferences:
        - "private-registry.io/*"
        imagePullSecrets:
          - name: registry-credentials

Issue 4: Performance Impact

Symptoms:

• Slow pod creation
• Webhook latency alerts

Diagnosis:

# Check Kyverno metrics
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/kyverno/pods

# Review policy execution time
kubectl logs -n kyverno -l app.kubernetes.io/name=kyverno | grep "processing time"

Solutions:

1. Use image digests instead of tags (faster verification)
2. Increase Kyverno replicas
3. Exclude system namespaces
4. Use background scanning for non-critical policies
5. Cache registry responses

Advanced Topics

1. Attestation Verification

Beyond signature verification, verify attestations:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-sbom-attestation
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: check-sbom-present
      match:
        any:
        - resources:
            kinds:
              - Pod
      verifyImages:
      - imageReferences:
        - "*"
        attestations:
        - predicateType: https://spdx.dev/Document
          attestors:
          - count: 1
            entries:
            - keys:
                publicKeys: |-
                  -----BEGIN PUBLIC KEY-----
                  [Your public key]
                  -----END PUBLIC KEY-----                  

2. Multi-Signature Requirements

Require multiple signatures:

verifyImages:
- imageReferences:
  - "critical-app/*"
  attestors:
  - count: 2  # Require 2 signatures
    entries:
    - keys:
        publicKeys: |-
          -----BEGIN PUBLIC KEY-----
          [Security team key]
          -----END PUBLIC KEY-----          
    - keys:
        publicKeys: |-
          -----BEGIN PUBLIC KEY-----
          [Platform team key]
          -----END PUBLIC KEY-----          

3. Time-Based Policies

Implement signing freshness requirements:

verifyImages:
- imageReferences:
  - "*"
  required: true
  mutateDigest: true
  verifyDigest: true
  attestors:
  - count: 1
    entries:
    - keys:
        publicKeys: |-
          [Public key]          
  attestations:
  - predicateType: https://slsa.dev/provenance/v0.2
    conditions:
    - all:
      - key: "{{ time_since('', '2006-01-02T15:04:05Z', '') }}"
        operator: LessThanOrEquals
        value: "24h"

4. Custom Image Registries with Authentication

For private registries requiring authentication:

apiVersion: v1
kind: Secret
metadata:
  name: registry-credentials
  namespace: kyverno
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64-encoded-config>
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-private-images
spec:
  validationFailureAction: Enforce
  rules:
    - name: verify-images
      verifyImages:
      - imageReferences:
        - "myregistry.io/*"
        imagePullSecrets:
          - name: registry-credentials
        attestors:
        - count: 1
          entries:
          - keys:
              publicKeys: |-
                [Public key]                

Conclusion

Implementing Cosign for container image signing combined with Kyverno for policy enforcement on AWS EKS creates a robust, secure software supply chain. This integration provides:

Key Takeaways

1. Security: Cryptographic assurance that only approved, unmodified images run in production
2. Compliance: Automated enforcement of security policies with audit trails
3. Automation: No manual intervention required for verification
4. Developer Experience: Seamless integration into existing CI/CD workflows
5. Operational Excellence: Centralized policy management across clusters
6. Risk Reduction: Significant decrease in supply chain attack surface

Implementation Checklist

• Install and configure Cosign in CI/CD pipeline
• Generate or provision signing keys (KMS recommended)
• Sign all container images as part of build process
• Deploy Kyverno to EKS cluster with HA configuration
• Create ClusterPolicies for image verification
• Test with signed and unsigned images
• Configure monitoring and alerting
• Document policies and procedures
• Train development teams
• Regular key rotation schedule
• Periodic policy review and updates

Next Steps

1. Start Small: Begin with a single application or namespace
2. Use Audit Mode: Run policies in audit mode initially to gather data
3. Gradual Rollout: Slowly expand to more critical workloads
4. Monitor Closely: Watch for false positives and adjust policies
5. Iterate: Continuously improve based on feedback and metrics

Resources

• Cosign Documentation: https://docs.sigstore.dev/cosign/overview
• Kyverno Documentation: https://kyverno.io/docs/
• Sigstore: https://www.sigstore.dev/
• SLSA Framework: https://slsa.dev/
• AWS EKS Best Practices: https://aws.github.io/aws-eks-best-practices/
• Kyverno Policies Library: https://kyverno.io/policies/

Community and Support

• Cosign GitHub: https://github.com/sigstore/cosign
• Kyverno GitHub: https://github.com/kyverno/kyverno
• Sigstore Slack: https://sigstore.slack.com
• Kyverno Slack: https://kubernetes.slack.com #kyverno

By implementing these tools and following the best practices outlined in this guide, you’ll establish a strong foundation for secure container deployments on AWS EKS, protecting your organization from supply chain attacks while maintaining developer velocity and operational excellence.