K3s in Production: A Field Report After Two Years

From Evaluation to Production

In April 2019, we installed K3s for the first time and were impressed by how quickly you could get a functioning Kubernetes cluster up and running. As we described in our K3s introductory article, the distribution was still young back then, the documentation sparse, and high availability was not officially supported. Our assessment at the time: excellent for development and CI, but too early for production.

Two years later, things look different. K3s has reached version 1.21, was acquired by SUSE (who acquired Rancher Labs in early 2020), is a CNCF Sandbox project, and has built a solid reputation as a certified Kubernetes distribution. And we now run several production workloads on it -- on nodes that would be too small for a full Kubernetes installation.

This post describes what we learned over those two years. Not as a rehash of the official documentation, but as an honest report from day-to-day operations.

Why Not Just Use Full Kubernetes?

The short answer: because it is simply overkill for certain workloads. Not every service needs an HA cluster with three etcd nodes, a separate control plane, and a worker pool. We run a number of internal tools and smaller client projects that need to run reliably but require neither traffic spikes handling nor horizontal scaling across dozens of pods.

For these workloads, we chose K3s -- deliberately. The arguments that convinced us:

Resource consumption. A full K8s control plane with etcd already consumes 1.5 to 2 GB of RAM on a small VPS at idle. K3s gets by with 400 to 500 MB. On nodes with 4 GB of RAM, that is the difference between "nothing else fits" and "three to four services run comfortably."

Operational complexity. K3s is a single binary under 70 MB. No etcd cluster that needs to be maintained separately. No complex certificate bootstrapping. No orchestration of five separate systemd services. This significantly reduces the attack surface for errors -- especially when, like us, you manage the infrastructure with a small team.

Full compatibility. K3s is certified Kubernetes. Our Helm charts, manifests, and CI/CD pipelines work identically. No vendor lock-in, no proprietary API. If a workload grows and needs a full cluster, we migrate -- without rewriting the deployments.

NixOS as the Host Operating System

A decision we have not regretted: our K3s nodes run on NixOS. It sounds like an unusual combination, but it works together excellently.

NixOS provides declarative system configuration. The entire state of a node -- installed packages, systemd services, firewall rules, kernel parameters -- is defined and versioned in a single Nix configuration file. When we need to set up a node from scratch, we deploy the configuration and have an identical state within minutes. No Ansible, no manual reconfiguration.

We do not install K3s itself via the official install script, but as a NixOS service. This makes the configuration reproducible and rollback-capable -- if an upgrade causes problems, we switch back to the previous NixOS generation. This gives us a safety net that we would not have with conventional Linux distributions.

SQLite vs etcd: The Datastore Question

K3s uses SQLite as the default datastore for cluster state. This was one of the points that made us hesitate about production use in 2019. Two years later, we can say: for single-server setups, SQLite works flawlessly.

The concerns you often hear -- that SQLite is not designed for concurrent writes, that performance degrades with larger clusters -- are valid, but they do not apply to our scenarios. We run single-server nodes with 15 to 30 pods. The write load on the datastore is minimal. In over two years, we have not had a single outage attributable to SQLite.

Where we would recommend etcd: for multi-server HA setups, which K3s has officially supported since version 1.19. Or for clusters with more than 50 nodes, where write load actually becomes relevant. For anything below that, SQLite is the simpler choice -- fewer moving parts, less maintenance overhead.

Traefik as the Bundled Ingress

K3s ships with Traefik as the ingress controller. In version 1.21, this is Traefik v1, which is starting to show its age. For our purposes, however, it works reliably: TLS termination with Let's Encrypt, automatic certificate renewal, clean ingress configuration via standard Kubernetes resources.

If you prefer Traefik v2 or another ingress controller like nginx-ingress, you can disable the bundled Traefik at startup:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik" sh -

After that, you install the desired controller via Helm just like in any other Kubernetes cluster. We did this on one of our nodes and are happy with Traefik v2. On the other nodes, the bundled Traefik v1 continues to run without issues. Once K3s switches to Traefik v2 as the default, we will replace the manually installed instances with the bundled version again.

Upgrades: Pragmatic Rather Than Elegant

Upgrades were initially the point that worried us most. With a full Kubernetes cluster using kubeadm, there is a defined upgrade path with kubeadm upgrade plan and kubeadm upgrade apply. With K3s, the approach is significantly simpler -- and that is by design.

A K3s upgrade on a single node looks like this:

# Aktuelle Version prüfen
k3s --version

# Upgrade auf eine neue Version via Install-Skript
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=v1.21 sh -

# Node-Status prüfen
sudo kubectl get nodes

The install script detects an existing installation and performs an in-place upgrade. The K3s process is restarted, the running pods are briefly interrupted and come back up afterward. For us, this typically means 30 to 60 seconds of downtime.

On our NixOS nodes, the process is even simpler: we update the K3s version in the Nix configuration and deploy with nixos-rebuild switch. Then we verify the state:

# Kubeconfig setzen
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

# Cluster-Status verifizieren
sudo kubectl get nodes -o wide

# Alle Pods prüfen
sudo kubectl get pods --all-namespaces

For services that cannot tolerate downtime, we move the workload to a second node beforehand. But honestly: most of our K3s workloads handle 60 seconds of interruption without any problem. That sounds like a compromise, and it is. But it is a deliberate compromise that drastically reduces operational overhead.

Resource Savings in Practice

The actual savings depend on the scenario, but here are the numbers from one of our nodes -- a VPS with 4 GB RAM and 2 vCPUs:

With K3s (current): The K3s process uses approximately 450 MB of RAM. Traefik, CoreDNS, and local-path-provisioner together add about 100 MB. That leaves a good 3.4 GB for workloads. We run four services in production on it.

Estimated with full K8s: etcd alone would have consumed 500 to 800 MB, plus kube-apiserver, kube-scheduler, kube-controller-manager, and kubelet separately. Realistically, 1.5 to 2 GB would have gone to the control plane. On a 4 GB node, that would have been enough for one to two services at most.

The difference is not academic. It determines whether you can use a 4-euro VPS for a small service or need a 15-euro VPS. With ten such nodes, that adds up.

What We Would Do Differently Today

Not everything went smoothly from the start. A few lessons we wish we had learned earlier:

Datastore backups. SQLite is a single file. We now back it up daily via a cron job. In the first few months, we did not do this -- negligent in retrospect.

# SQLite-Datastore sichern
sudo cp /var/lib/rancher/k3s/server/db/state.db /backup/k3s-state-$(date +%Y%m%d).db

Do not mix K3s and classic K8s in the same CI/CD pipeline without testing. K3s is compatible, but there are edge cases -- for instance with certain admission webhook configurations or when using cloud-provider-specific features that K3s has deliberately removed. We now test Helm charts against both variants.

Do not underestimate the Flannel network. K3s uses Flannel with VXLAN as the default CNI. For most scenarios, this works fine. But if you need network policies, you must install Calico or another CNI. Flannel alone does not support network policies -- a point to consider during planning.

Conclusion

Over two years, K3s has evolved from a promising experiment into a solid production solution. Not for every scenario -- if you need a large, highly available cluster with complex network policies and multi-tenancy, you are better served by full Kubernetes. But for the many cases where one or two nodes with a handful of services are sufficient, K3s is the more pragmatic choice.

The combination with NixOS as the host operating system has proven particularly valuable for us. Declarative configuration at the OS level, a lightweight Kubernetes on top, and standard Helm charts on that -- this creates a stack that can be operated with minimal effort while still providing the full Kubernetes API.

Anyone evaluating K3s today will find a much more mature distribution than what we tested in 2019. HA support is there, CNCF certification is there, the community has grown. The best way to get started is still the same: a Linux system, one command, thirty seconds of patience. Except now you can run the result in production with a clear conscience.