NixOS as a Server Operating System: Declarative Infrastructure

The Foundation Beneath the Containers

When people talk about Kubernetes, they talk about containers, pods, deployments, and services. What is surprisingly rarely discussed is the operating system on which all of this runs. Most Kubernetes setups are based on Ubuntu Server or Debian -- solid distributions that do their job reliably. But after several years of running on exactly these distributions, we realized that the host OS is the part of our infrastructure that causes us the most headaches.

Not because of instability. But because of a lack of reproducibility. When a server looks different after six months of operation than it did on the day it was set up -- because someone manually installed a package, because an apt upgrade overwrote a configuration file, because a service was activated that nobody can trace back to -- then you have a problem. Not today, but during the next debugging session at three in the morning.

For over a year now, we have been using NixOS as the host operating system for our Kubernetes nodes. In this post, we explain why we took this step and what has changed as a result.

What NixOS Does Differently

NixOS is a Linux distribution built on the functional package manager Nix. At first glance, this sounds like a niche for enthusiasts, but on closer inspection it is a fundamentally different approach to system configuration.

The key difference from Debian, Ubuntu, or CentOS: the entire state of a NixOS system is described in a single configuration file -- configuration.nix. Installed packages, enabled services, network configuration, firewall rules, users, SSH keys -- everything is declared in this file. There is no imperative apt install or systemctl enable. If something is not in the configuration, it does not exist on the system.

This has far-reaching consequences. If we copy the configuration.nix from Server A to a fresh Server B and run nixos-rebuild switch, we get an identical server. Not similar, not almost the same -- identical. The same packages in the same versions, the same services with the same settings.

Anyone who works with Infrastructure as Code recognizes the principle immediately: the configuration file is the single source of truth. The difference from Ansible or Puppet is that NixOS does not aim for convergence but guarantees an exact state. There is no drift, because the system simply cannot assume any state other than the configured one.

Our K3s Node Configuration

We use NixOS exclusively as the host operating system. Our applications do not run as Nix packages but as containers on Kubernetes. NixOS is the foundation, K3s runs on top of it, and within K3s our workloads run as Docker containers. This separation is important to us: Nix is excellent for declarative system configuration, but our application deployment stays with Kubernetes and Helm -- the tools we have described in detail in previous posts.

A typical configuration.nix for one of our K3s nodes looks roughly like this:

{ config, pkgs, ... }:

{
  # Netzwerk-Konfiguration
  networking.hostName = "k3s-node-01";
  networking.interfaces.ens3.ipv4.addresses = [{
    address = "10.0.1.10";
    prefixLength = 24;
  }];
  networking.defaultGateway = "10.0.1.1";
  networking.nameservers = [ "10.0.1.1" ];

  # Firewall: K3s-relevante Ports freigeben
  networking.firewall = {
    enable = true;
    allowedTCPPorts = [ 22 6443 10250 ];
    allowedUDPPorts = [ 8472 ];
  };

  # SSH-Zugang
  services.openssh = {
    enable = true;
    settings = {
      PermitRootLogin = "no";
      PasswordAuthentication = false;
    };
  };

  # K3s als Service aktivieren
  services.k3s = {
    enable = true;
    role = "agent";
    serverAddr = "https://10.0.1.5:6443";
    tokenFile = "/etc/k3s/token";
  };

  # Grundlegende Pakete auf dem Host
  environment.systemPackages = with pkgs; [
    vim
    htop
    curl
    kubectl
  ];

  # Benutzer
  users.users.deploy = {
    isNormalUser = true;
    extraGroups = [ "wheel" ];
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAA... deploy@encircle360"
    ];
  };

  # NixOS-Version pinnen
  system.stateVersion = "21.11";
}

That is the complete configuration of a K3s worker node. Network, firewall, SSH, K3s service, installed packages, users -- all in one file. Even someone who has never seen NixOS can read this configuration and understand what runs on the server. This is a significant advantage over the combination of Bash scripts, Ansible playbooks, and manual adjustments we had before.

The current NixOS version 21.11 ("Porcupine") that we use ships with K3s as a built-in module. This was not always the case -- in earlier versions, you had to configure K3s manually as a systemd service. The fact that the NixOS community has adopted K3s as a first-class module shows how seriously the distribution is taken in the server space.

Atomic Upgrades and Generations

The second major advantage of NixOS, alongside declarative configuration, is atomic upgrades. When we make a change to configuration.nix and run nixos-rebuild switch, the following happens: Nix calculates the new system state, downloads the required packages, assembles the configuration -- and activates the new state atomically. There is no in-between state where the system is half old and half new.

# Konfiguration anpassen
sudo vim /etc/nixos/configuration.nix

# Neuen Systemzustand anwenden
sudo nixos-rebuild switch

Each nixos-rebuild switch creates a new so-called generation. The system retains previous generations, and each generation is a complete, bootable system state. In the GRUB boot menu, you can see recent generations as separate entries.

This means: if an upgrade goes wrong -- a kernel update causes problems, a service no longer starts, whatever -- we simply boot the previous generation. No manual downgrade, no searching for the old package version, no hoping that apt can restore the old state. A reboot with the previous generation, and the server runs as before.

# Zur vorherigen Generation zurückkehren
sudo nixos-rebuild switch --rollback

# Alle verfügbaren Generationen auflisten
sudo nix-env --list-generations --profile /nix/var/nix/profiles/system

In practice, we have used this rollback three times so far. Each time, it was a process of under two minutes. On our Ubuntu servers, diagnosis alone would have taken longer.

Comparison with Ubuntu and Debian

We did not choose NixOS because Ubuntu or Debian are bad. Both are proven server distributions with excellent documentation and a huge community. But they take a fundamentally different approach.

State management. Ubuntu and Debian manage their state imperatively. You install packages, enable services, edit configuration files -- and the current state is the result of all past actions. If you want to know after a year why a particular package is installed, you have to search logs or guess. With NixOS, the answer is always in configuration.nix.

Reproducibility. Reproducing an Ubuntu server exactly requires external tools like Ansible, Puppet, or Terraform. With NixOS, reproducibility is built in. The configuration file is simultaneously the documentation and the deployment script.

Upgrades. apt upgrade on an Ubuntu server might work -- or it might not. Partially updated systems, dependency conflicts, overwritten configuration files -- anyone who has run Ubuntu servers for more than a year knows these problems. NixOS upgrades are atomic and reversible.

However, NixOS comes at a cost: the learning curve is steep. The Nix language takes getting used to, the documentation has improved but is still not at the level of Debian's or Ubuntu's wiki. And when you have a problem that goes beyond the standard configuration, experience reports online are harder to find. We certainly invested time upfront to understand the Nix ecosystem. That investment has paid off, but it was real.

Why Not Nix for Everything?

A question that keeps coming up: if NixOS is so good, why don't our applications also run as Nix packages? Why the detour through Kubernetes and containers?

The answer is pragmatic. NixOS is excellent as a host operating system: stable, reproducible, low-maintenance. But our application deployment workflow is based on Docker images, Helm charts, and Kubernetes manifests. Our developers build containers, our CI pipeline produces containers, our clients understand containers. The entire tooling -- monitoring, logging, secrets management, networking -- is designed for Kubernetes.

NixOS as the host OS and Kubernetes as the application platform -- this is not a compromise but a deliberate separation of layers. Each layer does what it does best. NixOS ensures that the host is reproducible, up to date, and stable. Kubernetes ensures that applications scale, get updated, and run with high availability.

Conclusion

NixOS has noticeably improved our server infrastructure. Declarative configuration eliminates configuration drift. Atomic upgrades and generations make updates safe and reversible. Reproducibility allows us to set up new nodes in minutes rather than hours.

The path there was not without effort. Learning Nix takes time, and not every problem can be solved with a Stack Overflow search. But for the specific use case -- server operating system for Kubernetes nodes -- the advantages clearly outweigh the costs. Our nodes are predictable, our upgrades are painless, and if something does go wrong, we are one reboot away from the previous working version.

Anyone who runs Kubernetes on their own hardware and struggles with maintaining the host operating system should seriously consider NixOS. The learning curve is real, but the result is an infrastructure that behaves the way you would expect from Infrastructure as Code -- not just at the Kubernetes level, but all the way down to the operating system.