Helmfile for Multi-Cluster Deployments: Environments and Selectors

From One Cluster to Three

When we added Helmfile to our stack nearly three years ago, the starting point was straightforward: one cluster, a handful of releases, one helmfile.yaml. It worked beautifully -- declarative Helm management, environment-specific values, a single command for the entire deployment. At the time, we described how Helmfile replaces the imperative approach of individual helm upgrade commands with a declarative YAML file.

By now, at encircle360, we operate three K3s clusters: Development, Staging, and Production. Each cluster runs on its own nodes, as we described in our experience report on K3s in production use. The clusters share most of their releases -- the same Spring Boot services, the same ingress controller, the same monitoring -- but the configurations differ. Different domains, different resource limits, different replica counts, different secrets.

The question was: how do we manage three clusters with a single Helmfile configuration without ending up in copy-and-paste chaos?

Environments as the Foundation

Helmfile comes with the concept of environments built in. In our first Helmfile article, we briefly mentioned them. For multi-cluster setups, they are the central tool.

An environment in Helmfile corresponds to a deployment target -- in our case, a cluster. Each environment brings its own global variables that are available in all release definitions.

# helmfile.yaml

environments:
  dev:
    values:
      - environments/dev.yaml
    kubeContext: k3s-dev
  staging:
    values:
      - environments/staging.yaml
    kubeContext: k3s-staging
  production:
    values:
      - environments/production.yaml
    kubeContext: k3s-production

The kubeContext directive is crucial. It ensures that Helmfile automatically uses the correct kubectl context when an environment is selected. No manual kubectl config use-context before each deployment. A helmfile -e production apply always targets the production cluster, regardless of which context is currently active.

The environment values files define global variables that differ between clusters:

# environments/production.yaml

clusterName: production
domain: encircle360.com
registry: registry.encircle360.com
ingressClass: traefik
certManager:
  issuer: letsencrypt-prod
resources:
  defaultCpuRequest: 250m
  defaultMemoryRequest: 256Mi
  defaultCpuLimit: 500m
  defaultMemoryLimit: 512Mi
replicas:
  default: 2

# environments/dev.yaml

clusterName: dev
domain: dev.encircle360.com
registry: registry.encircle360.com
ingressClass: traefik
certManager:
  issuer: letsencrypt-staging
resources:
  defaultCpuRequest: 100m
  defaultMemoryRequest: 128Mi
  defaultCpuLimit: 250m
  defaultMemoryLimit: 256Mi
replicas:
  default: 1

These values are then available as Go template variables in the release definitions. Instead of maintaining separate values files per cluster, the templates reference the global variables:

releases:
  - name: api-service
    namespace: application
    chart: ./charts/spring-boot-app
    version: 2.3.0
    values:
      - values/api-service.yaml
      - values/{{ .Environment.Name }}/api-service.yaml

The first values file contains the cross-cluster defaults. The second -- optional -- contains environment-specific overrides. This keeps the configuration DRY while clearly separating specific customizations.

Selectors for Targeted Deployments

Not every deployment requires updating all releases. When the monitoring stack is running stably and only a single service needs an update, a full helmfile apply would be overkill. Helmfile solves this with labels and selectors.

Each release can carry arbitrary labels:

releases:
  - name: api-service
    namespace: application
    chart: ./charts/spring-boot-app
    labels:
      tier: application
      team: backend
    values:
      - values/api-service.yaml

  - name: web-frontend
    namespace: application
    chart: ./charts/angular-app
    labels:
      tier: application
      team: frontend
    values:
      - values/web-frontend.yaml

  - name: prometheus
    namespace: monitoring
    chart: prometheus-community/kube-prometheus-stack
    labels:
      tier: infrastructure
      team: platform
    values:
      - values/monitoring.yaml

  - name: ingress-nginx
    namespace: ingress
    chart: ingress-nginx/ingress-nginx
    labels:
      tier: infrastructure
      team: platform
    values:
      - values/ingress.yaml

Using the --selector parameter (or -l for short), you filter at execution time:

# Deploy only the application tier
helmfile -e production -l tier=application apply

# Only releases from the backend team
helmfile -e staging -l team=backend apply

# Only a single release
helmfile -e dev -l name=api-service apply

# Combination: infrastructure tier in staging
helmfile -e staging -l tier=infrastructure diff

In our daily work, we use selectors constantly. A typical workflow looks like this: the backend team updates their service, pushes the chart version and values change to the repository, and the pipeline runs helmfile -e staging -l team=backend apply. The rest of the cluster remains untouched.

helmfile diff: Reviewing Changes Before Deployment

Before deploying to production, we use helmfile diff as a mandatory step. The command compares the desired state from the Helmfile configuration with the actual state in the cluster and displays the differences as a color-coded diff.

# Full diff against production
helmfile -e production diff

# Diff only for application releases
helmfile -e production -l tier=application diff

The output resembles a git diff at the Kubernetes manifest level. You see exactly which fields change -- new environment variables, modified resource limits, updated image tags. In our CI/CD pipelines, helmfile diff runs as its own job, with output visible in the pipeline log. This gives the team the opportunity to review the changes before helmfile apply actually applies them.

For the production cluster, this is not optional. We have experienced too often that a seemingly harmless values change had unexpected side effects -- a missing value that became an empty string, a changed port that broke the health check. helmfile diff catches this before it reaches the cluster.

Secrets Management with vals

Secrets do not belong in plaintext in a Git repository. For Helmfile, there are several approaches, and we chose vals -- a tool that injects secrets at runtime from external sources.

vals supports various backends: SOPS-encrypted files, HashiCorp Vault, AWS Secrets Manager, and others. In our setup, we use SOPS with age as the encryption tool. The encrypted secrets reside in the repository, and vals decrypts them during helmfile apply.

# values/production/secrets.yaml (encrypted with SOPS)

databasePassword: ENC[AES256_GCM,data:abc123...,type:str]
apiKey: ENC[AES256_GCM,data:def456...,type:str]

In the helmfile.yaml, you activate vals as the secrets backend:

helmDefaults:
  wait: true
  timeout: 300

environments:
  production:
    secrets:
      - environments/production-secrets.yaml

The advantage over the older helm-secrets plugin: vals is independent of Helm and can be used outside of Helmfile as well. The secrets reside encrypted in the repository, go through the normal code review process, and are never visible in plaintext -- except during deployment, where vals decrypts them and passes them to Helm.

Directory Structure for Multi-Cluster

With three clusters, dozens of releases, and environment-specific values, the directory structure grows. We tried several approaches and settled on the following structure:

helmfile/
  helmfile.yaml
  environments/
    dev.yaml
    dev-secrets.yaml
    staging.yaml
    staging-secrets.yaml
    production.yaml
    production-secrets.yaml
  values/
    api-service.yaml          # Cross-cluster defaults
    web-frontend.yaml
    monitoring.yaml
    ingress.yaml
    dev/
      api-service.yaml        # Dev-specific overrides
    staging/
      api-service.yaml
    production/
      api-service.yaml
      web-frontend.yaml
  charts/
    spring-boot-app/
    angular-app/
  .sops.yaml                  # SOPS configuration

The basic idea: defaults go in values/<release>.yaml, environment-specific deviations in values/<env>/<release>.yaml. Not every release needs environment-specific values -- if the defaults are sufficient, the file simply does not exist. Helmfile handles this gracefully when you set missingFileHandler: Warn.

helmDefaults:
  missingFileHandler: Warn

For larger setups, Helmfile also supports splitting into multiple files with helmfiles:

helmfiles:
  - path: helmfile.d/infrastructure.yaml
  - path: helmfile.d/applications.yaml
  - path: helmfile.d/monitoring.yaml

Each sub-file contains its own releases and can be deployed independently. We do not use this yet, as our overall configuration with around thirty releases is still manageable in a single file. But the option is ready when complexity continues to grow.

CI/CD Integration: Three Clusters, One Pipeline

The pipeline integration follows a clear pattern. Each cluster has its own deployment stage, but all use the same Helmfile configuration:

# Development: Automatically on every push to develop
helmfile -e dev apply

# Staging: Automatically on merge to main
helmfile -e staging apply

# Production: Manually triggered after staging validation
helmfile -e production diff    # Review step
helmfile -e production apply   # After approval

The kubectl context is set in the pipeline via kubeconfig. Each cluster has its own service account with restricted permissions. The pipeline needs no logic to distinguish between clusters -- Helmfile handles this through environments.

Conclusion

Helmfile scales from a single cluster to a multi-cluster landscape without requiring changes to the fundamental concept. Environments represent clusters, selectors enable targeted deployments, and vals keeps secrets secure. The directory structure with cross-cluster defaults and environment-specific overrides avoids duplication without losing clarity.

Anyone already using Helmfile for a single cluster -- as we described in our introductory article -- can take the step to multiple clusters incrementally. First define environments, then split values, then introduce selectors. Each step provides immediate value and requires no restructuring of the existing configuration.

In our projects, this setup has proven itself over the past months. Three K3s clusters, around thirty releases per cluster, one Git repository as the single source of truth. Changes go through pull requests, helmfile diff shows the impact, and helmfile apply brings the desired state to the cluster. Declarative, traceable, and -- with some structure -- manageable even across multiple clusters.