Kyverno in Production: Lessons Learned After Two Years

Two Years Later

At the end of 2021, we described in our Kyverno introductory article why we chose Kyverno as our policy engine. Back then, Kyverno was a CNCF Sandbox project at version 1.5. Today, just over two years later, Kyverno is a CNCF Graduated Project -- the same maturity level as Kubernetes itself -- and we're running version 1.11. The policy engine has established itself as a core part of our cluster infrastructure, and we've learned a great deal along the way.

This article summarizes our experiences: what worked, where we made mistakes, and which policies we now consider indispensable.

Audit vs. Enforce: The Right Strategy

The most important lesson of the first few months was: Never roll out a new policy directly in Enforce mode. We did this exactly once -- a policy that was supposed to block missing resource limits -- and ended up blocking a deployment on Monday morning because an internal tool was configured without limits. The team was not amused.

Since then, we follow a strict three-step plan:

1. Deploy in Audit mode -- the policy runs but doesn't block anything. Kyverno creates PolicyReports for every violation.
2. Evaluate reports and fix violations -- typically over two to four weeks, depending on the policy's scope.
3. Switch to Enforce -- only when no violations remain.

In YAML, the migration from Audit to Enforce looks trivial:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
  annotations:
    policies.kyverno.io/title: Require Resource Limits
    policies.kyverno.io/severity: medium
spec:
  # Step 1: Audit
  # validationFailureAction: Audit
  # Step 3: Enforce
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-limits
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: >-
          All containers must define CPU and memory limits.
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"

The key is the discipline to actually stick to the audit period. It's tempting to immediately enforce an obviously sensible policy. But in a cluster with dozens of services and multiple teams, there are always exceptions you hadn't thought of.

Policy Exceptions: Rules with Exceptions

Starting with Kyverno 1.9, there's a dedicated feature for Policy Exceptions -- and it has fundamentally improved how we handle special cases. Previously, we built exceptions directly into the policy, which made policies hard to read. Now exceptions are standalone resources:

apiVersion: kyverno.io/v2beta1
kind: PolicyException
metadata:
  name: allow-monitoring-without-limits
  namespace: monitoring
spec:
  exceptions:
    - policyName: require-resource-limits
      ruleNames:
        - check-limits
  match:
    any:
      - resources:
          kinds:
            - Pod
          namespaces:
            - monitoring
          names:
            - "prometheus-node-exporter-*"

This exception allows the Prometheus Node Exporter to run without resource limits -- which can make sense for DaemonSets on nodes with varying hardware specs. The exception is clearly documented, lives in the Git repository, and goes through the same review process as any other change.

Our principle: every exception needs a comment explaining why it exists. Exceptions without justification are rejected in review.

Policy Testing with the Kyverno CLI

In the Kyverno introductory article, we had announced our intention to integrate policies into the CI pipeline. We've since implemented this, and it was one of the best decisions in the entire setup.

The Kyverno CLI offers a test command that validates policies against defined test cases -- locally, without a running cluster. The test definition is a YAML file:

# tests/require-resource-limits/kyverno-test.yaml
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: test-require-resource-limits
policies:
  - ../../policies/require-resource-limits.yaml
resources:
  - resources.yaml
results:
  - policy: require-resource-limits
    rule: check-limits
    resource: pod-with-limits
    kind: Pod
    result: pass
  - policy: require-resource-limits
    rule: check-limits
    resource: pod-without-limits
    kind: Pod
    result: fail

Along with the test resources:

# tests/require-resource-limits/resources.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-limits
spec:
  containers:
    - name: app
      image: nginx:1.25
      resources:
        limits:
          cpu: 500m
          memory: 256Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: pod-without-limits
spec:
  containers:
    - name: app
      image: nginx:1.25

Running kyverno test tests/require-resource-limits/ validates that the policy accepts the pod with limits and rejects the one without. This runs in our CI pipeline on every push that modifies policy files. Since we introduced this, we haven't had a single case of a faulty policy making it into the cluster.

CI/CD Integration

The Kyverno CLI does more than just testing. We also use it to validate our Helm Charts against the active policies before they're deployed. The pipeline workflow looks like this:

# 1. Render Helm templates
helm template my-release ./charts/my-app \
  -f values/production.yaml > rendered.yaml

# 2. Validate against Kyverno policies
kyverno apply policies/ --resource rendered.yaml

# 3. Deploy only on success
helmfile -e production apply

The second step fails if the rendered manifest would violate an Enforce policy. This is a crucial advantage over the pure admission webhook approach: errors are caught early, not just at deployment time. Combined with our Helmfile multi-cluster setup, we validate the manifests for all three clusters in a single pipeline job.

Performance: What We Observed

Kyverno runs as an admission webhook and therefore sits in the critical path of every API request. Performance is not a theoretical problem -- it's a real one. Our experiences:

Memory consumption: Kyverno keeps policy state and cached resources in memory. With roughly 25 ClusterPolicies and 10 PolicyExceptions, our Kyverno pod sits at about 350 MB RSS. That's acceptable, but worth keeping an eye on. With more than 50 policies or policies with complex pattern matches, consumption increases noticeably.

Latency: On average, Kyverno adds 10-30ms to API response time. For regular deployments, this is irrelevant. For batch operations -- such as when a Helmfile apply updates thirty releases simultaneously -- this can add up. We've kept the webhook timeout configuration at 15 seconds and never had a timeout issue.

Replicas: Since Kyverno 1.9, the project recommends three replicas for production. We run two replicas with a PodDisruptionBudget of minAvailable: 1. This is sufficient for our cluster size and survives node maintenance without outages.

Must-Have Policies: Our Baseline Set

After two years, five policies have emerged that we deploy on every cluster -- they form the minimum for a cleanly operated cluster:

1. Require resource limits -- prevents containers from being deployed without CPU and memory limits. Without limits, a single pod can destabilize the entire node.

2. Disallow latest tag -- as described in the introductory article. The latest tag is non-deterministic and has no place in production.

3. Enforce standard labels -- every deployment must carry app.kubernetes.io/name, app.kubernetes.io/version, and a team label. This is the foundation for usable monitoring and alerting.

4. Restrict image registries -- containers may only pull images from our private registry and explicitly approved public registries. This prevents unvetted images from accidentally entering the cluster.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  rules:
    - name: validate-registries
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: >-
          Images may only come from approved registries.
        pattern:
          spec:
            containers:
              - image: "registry.encircle360.com/* | docker.io/library/* | ghcr.io/*"

5. Generate default NetworkPolicy -- every new namespace automatically gets a deny-all ingress policy. This was already described in the introductory article and remains one of our most valuable generate policies.

These five policies catch the most common configuration errors and establish a baseline that applies to all teams. Everything beyond that -- more specific validations, compliance requirements, team-specific rules -- we add as needed.

Policy Organization in the Repository

Our policies live in the same Git repository as the Helmfile configuration. The directory structure:

kyverno/
  policies/
    require-resource-limits.yaml
    disallow-latest-tag.yaml
    require-labels.yaml
    restrict-image-registries.yaml
    generate-default-networkpolicy.yaml
  exceptions/
    monitoring-limits-exception.yaml
  tests/
    require-resource-limits/
      kyverno-test.yaml
      resources.yaml
    disallow-latest-tag/
      kyverno-test.yaml
      resources.yaml

The policies are deployed via a Kyverno release in our Helmfile. Changes to policies go through the same pull request workflow as any other infrastructure change: branch, CI tests, review, merge, deploy.

Conclusion

Over two years, Kyverno has evolved from a promising sandbox project to a mature, CNCF-graduated tool -- and in our infrastructure, from an experiment to a load-bearing pillar. The combination of YAML-based policies, the CLI testing framework, and seamless integration into existing Kubernetes workflows makes it the ideal policy engine for teams our size.

The key lessons learned in brief: always Audit first, then Enforce. Test policies like code. Manage exceptions explicitly and with documentation. Keep an eye on performance, but don't over-engineer. And start with a solid baseline set rather than trying to regulate everything at once.

If you're not yet using Kyverno and you're running Kubernetes in production, we highly recommend getting started. The learning curve is gentle, the benefits are immediately tangible -- and after two years, we can say: it delivers on its promises.