GitOps with Helmfile and Kyverno: Our Deployment Workflow

Two Tools, One Workflow

Over the past few years, we have described in separate articles how we use Helmfile for declarative Helm management, how we use Kyverno as a policy engine, and how both have proven themselves in larger setups -- from multi-cluster deployments with Helmfile to Kyverno in production. What was missing until now was the article showing how these building blocks work together in our daily deployment workflow.

The core of our approach is simple: Git is the single source of truth. Every change to the infrastructure -- whether a new deployment, a modified configuration, or an updated policy -- goes through a pull request, is validated by Kyverno, and applied to the clusters by Helmfile. No manual kubectl apply, no helm install on the command line.

Git as Single Source of Truth

Our infrastructure repository has been the central place for everything running on our Kubernetes clusters since mid-2023. The directory structure follows the pattern we described in the multi-cluster article, extended with a dedicated directory for Kyverno policies:

infra/
├── helmfile.yaml
├── environments/
│   ├── dev/
│   │   ├── values.yaml
│   │   └── secrets.yaml.enc
│   ├── staging/
│   │   └── values.yaml
│   └── prod/
│       ├── values.yaml
│       └── secrets.yaml.enc
├── policies/
│   ├── base/
│   │   ├── require-labels.yaml
│   │   ├── restrict-image-registries.yaml
│   │   └── disallow-privileged.yaml
│   └── prod/
│       ├── require-resource-limits.yaml
│       └── restrict-nodeport.yaml
└── releases/
    ├── cert-manager.yaml
    ├── ingress-nginx.yaml
    └── kyverno.yaml

Every change to this repository creates a pull request. Nobody pushes directly to main. This is not a convention but enforced through branch protection rules in Gitea.

Helmfile with Integrated Kyverno

Our helmfile.yaml manages Kyverno as a regular Helm release -- just like any other infrastructure service. This ensures that Kyverno itself goes through the same declarative lifecycle as everything else:

repositories:
  - name: kyverno
    url: https://kyverno.github.io/kyverno/

environments:
  dev:
    values:
      - environments/dev/values.yaml
  staging:
    values:
      - environments/staging/values.yaml
  prod:
    values:
      - environments/prod/values.yaml
    secrets:
      - environments/prod/secrets.yaml.enc

releases:
  - name: kyverno
    namespace: kyverno
    chart: kyverno/kyverno
    version: 3.3.4
    values:
      - releases/kyverno.yaml
      - replicaCount: {{ .Values | getOrNil "kyverno.replicas" | default 1 }}

  - name: kyverno-policies
    namespace: kyverno
    chart: kyverno/kyverno-policies
    version: 3.3.4
    needs:
      - kyverno/kyverno
    values:
      - validationFailureAction: {{ .Values | getOrNil "kyverno.validationAction" | default "Audit" }}

The crucial point is the environment-dependent validationFailureAction. In Dev, it is set to Audit -- violations are logged but not blocked. In Staging and Production, it is set to Enforce. This gives developers the freedom to experiment in Dev while policies are strictly enforced in higher environments.

Additionally, we deploy our own policies as separate Kyverno resources. Their YAML files live in the policies/ directory and are applied through a dedicated Helmfile release that uses a local chart.

Pre-Deploy Policy Checks in CI

The real strength of the workflow lies not in the cluster but in the CI pipeline. Before Helmfile applies anything, the Kyverno CLI checks all generated manifests against our policies. This catches violations before they reach the cluster.

Our Gitea Actions pipeline for the infrastructure repository looks like this:

name: Deploy Infrastructure
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install tools
        run: |
          # Helmfile
          curl -fsSL -o helmfile.tar.gz \
            https://github.com/helmfile/helmfile/releases/download/v0.169.2/helmfile_0.169.2_linux_amd64.tar.gz
          tar xzf helmfile.tar.gz && mv helmfile /usr/local/bin/
          # Kyverno CLI
          curl -fsSL -o kyverno-cli.tar.gz \
            https://github.com/kyverno/kyverno/releases/download/v1.13.2/kyverno-cli_v1.13.2_linux_x86_64.tar.gz
          tar xzf kyverno-cli.tar.gz && mv kyverno /usr/local/bin/kyverno-cli

      - name: Template manifests
        run: |
          helmfile -e ${{ env.TARGET_ENV }} template > /tmp/rendered-manifests.yaml

      - name: Validate against policies
        run: |
          kyverno-cli apply policies/base/ \
            --resource /tmp/rendered-manifests.yaml \
            --detailed-results
          if [ "${{ env.TARGET_ENV }}" = "prod" ]; then
            kyverno-cli apply policies/prod/ \
              --resource /tmp/rendered-manifests.yaml \
              --detailed-results
          fi

  diff:
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request'
    needs: validate
    steps:
      - uses: actions/checkout@v4

      - name: Helmfile diff
        run: |
          helmfile -e ${{ env.TARGET_ENV }} diff --context 3
        env:
          KUBECONFIG: ${{ secrets.KUBECONFIG }}

  apply:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    needs: validate
    steps:
      - uses: actions/checkout@v4

      - name: Helmfile apply
        run: |
          helmfile -e ${{ env.TARGET_ENV }} apply --context 3
        env:
          KUBECONFIG: ${{ secrets.KUBECONFIG }}

The flow has three stages. First, helmfile template renders all manifests without applying them. Then kyverno-cli apply checks these manifests against our policies -- completely offline, without cluster access. Only when validation passes does the pipeline execute either helmfile diff (for pull requests) or helmfile apply (for merges to main).

Policy Violations in Pull Requests

When the Kyverno validation fails, the developer sees it directly in the pull request. The CI pipeline fails, and the output shows precisely which policy was violated and in which manifest:

Policy: require-labels
Rule: check-team-label
Result: FAIL
Resource: apps/v1/Deployment/api-gateway
Message: label 'app.kubernetes.io/team' is required

This is a deliberate shift-left approach. Instead of Kyverno rejecting the admission request in the cluster and the developer only learning about it during the deployment attempt, the PR check fails minutes after the push. The feedback loop is short, and the fix happens in the same PR.

In practice, most violations occur in the first few weeks after introducing a new policy. After that, the team has internalized the rules, and the CI checks serve more as a safety net than a hurdle.

Environment Promotion: From Dev to Production

Our promotion model follows a clear path: Dev, then Staging, then Production. What changes is not the manifests themselves but the environment-specific values in the Helmfile environments.

A typical flow for a new service deployment looks like this:

1. Dev: The developer creates a PR that adds the new release to helmfile.yaml and configures Dev values. CI renders and validates. After the merge, the pipeline deploys to the Dev cluster.

2. Staging: A second PR adds the Staging values or updates the image version. The same validation runs, this time with Enforce policies. After the merge, the release goes to Staging.

3. Production: A third PR with the Prod values. Additionally, the Prod-specific policies take effect -- for example, the requirement that resource limits must be set or that only images from our internal registry are allowed.

Each step is a separate, reviewable PR. There is no automatic promotion mechanism from Dev to Staging. This is intentional: we want a human to explicitly trigger and confirm the transition. Automation within an environment yes, automatic promotion between environments no.

Kyverno as the Last Line of Defense

Even though CI validation catches most violations, Kyverno runs in the cluster in Enforce mode as the last line of defense. This is relevant for two scenarios: first, when someone does apply something manually to the cluster -- which can happen in emergency situations. Second, when policies change after manifests have already been validated and merged.

As described in our article on Kyverno in production, we use policy exceptions for exemptions that are technically justified and time-limited. These exceptions also live in the Git repository and go through the same review process.

Lessons Learned After One Year

We have been using this workflow in its current form since early 2024. Previously, we had Helmfile and Kyverno in use separately -- we built this up incrementally starting from the early days with Helm 3 -- but integrating them into a coherent GitOps pipeline was a project in itself.

The key takeaways:

Helmfile template is the key. The ability to render all manifests without applying them makes the entire pre-validation possible. Without this step, you would have to rely on cluster-side admission control, and the feedback loop would be significantly longer.

Policy versions belong in the Helmfile. We pin the Kyverno chart version just like any other dependency. An unintended policy update that suddenly blocks deployments is the last thing you want in production.

The Kyverno CLI must be the same version as the cluster. We had a case where the CLI version was newer than the installed Kyverno and accepted a policy syntax that the admission controller in the cluster did not yet understand. Since then, we pin both versions together.

Fewer policies, more strictly enforced. We initially introduced too many policies at once. This led to a flood of CI errors and reduced acceptance on the team. Now we introduce new policies in Audit mode first, communicate them, and switch them to Enforce after two to four weeks.

Helmfile vs. Argo CD

We do not want to hide the fact that in some newer setups we use Argo CD instead of Helmfile. Argo CD brings a pull-based GitOps approach: it continuously monitors the Git repository and automatically synchronizes the cluster state. This has advantages -- particularly the immediate detection of drift between Git and the cluster. The Helmfile workflow described here is push-based: the CI pipeline actively applies changes.

Both approaches work well with Kyverno. Argo CD synchronizes manifests through the Kubernetes API server, and Kyverno validates them there as an admission controller -- regardless of where the manifest came from. For us, the choice between Helmfile and Argo CD is not a matter of ideology but a project-specific decision. Helmfile shines with existing setups that have many Helm charts and complex values hierarchies. Argo CD is better suited for environments where continuous synchronization and a graphical overview are desired.

Conclusion

The workflow is not a complicated setup. It is ultimately two tools -- Helmfile (or alternatively Argo CD) and Kyverno -- connected through Git as the central source of truth. The real work lies in maintaining discipline: everything goes through Git, everything is validated, and manual interventions remain the documented exception. This is not a perfect system, but it is one we understand, can maintain, and that works reliably.