NixOS in Practice: Reproducible Server Configuration

Two Years Later

In February 2022, we described in our NixOS introductory article why we chose NixOS as the host operating system for our Kubernetes nodes. The core principles -- declarative configuration, atomic upgrades, rollbacks via generations -- haven't changed since then. What has changed is how we use NixOS in our daily work.

Back then, we had a handful of nodes, each with its own configuration.nix that we copied to the respective server via SSH and applied with nixos-rebuild switch. It worked, but it was neither scalable nor reproducible in the strict sense. The nixpkgs version wasn't pinned, configuration was shared between nodes via copy-paste, and deployment was a manual process.

Today, with NixOS 23.05 on our production servers and the upcoming upgrade to 23.11, our workflow looks fundamentally different. The key to this is Nix Flakes.

Nix Flakes: Taking Reproducibility Seriously

Flakes are the feature that took NixOS from "good" to "indispensable" for us. The concept is simple: A flake.nix file defines explicit inputs -- typically the nixpkgs version -- and explicit outputs -- in our case, the configurations of our servers. Each input is pinned to an exact Git commit via a flake.lock file.

This means: If we deploy a configuration today and check out the same commit six months later and deploy again, we get exactly the same system. Not similar, not "should be fine" -- identical. No surprisingly updated packages, no shifted dependencies.

Our flake.nix looks roughly like this:

{
  description = "encircle360 server infrastructure";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-23.05";
  };

  outputs = { self, nixpkgs }: {
    nixosConfigurations = {
      k3s-server-01 = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          ./hosts/k3s-server-01/configuration.nix
          ./modules/base.nix
          ./modules/k3s-node.nix
        ];
      };

      k3s-worker-01 = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          ./hosts/k3s-worker-01/configuration.nix
          ./modules/base.nix
          ./modules/k3s-node.nix
        ];
      };

      k3s-worker-02 = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          ./hosts/k3s-worker-02/configuration.nix
          ./modules/base.nix
          ./modules/k3s-node.nix
        ];
      };
    };
  };
}

All servers are defined in a single flake.nix. Each server has its host-specific configuration but shares common modules. The flake.lock pins nixpkgs to a specific commit, and a nix flake update updates this reference intentionally -- not differently with every build, but only when we explicitly want it.

Shared Modules: Configuration Without Copy-Paste

The biggest practical benefit of Flakes is clean modularization. Previously, we copied identical configuration blocks between nodes -- SSH settings, firewall rules, K3s configuration. When something changed, we had to touch each node individually. This was error-prone and contradicted the entire idea of declarative configuration.

Today, we extract shared configuration into reusable modules. Our base.nix module contains everything every server needs:

# modules/base.nix
{ config, pkgs, ... }:

{
  # SSH hardening
  services.openssh = {
    enable = true;
    settings = {
      PermitRootLogin = "no";
      PasswordAuthentication = false;
      KbdInteractiveAuthentication = false;
    };
  };

  # Essential packages on the host
  environment.systemPackages = with pkgs; [
    vim
    htop
    curl
    tmux
    git
  ];

  # Automatic garbage collection
  nix.gc = {
    automatic = true;
    dates = "weekly";
    options = "--delete-older-than 30d";
  };

  # Enable Flakes
  nix.settings.experimental-features = [ "nix-command" "flakes" ];

  # Deploy user
  users.users.deploy = {
    isNormalUser = true;
    extraGroups = [ "wheel" ];
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAA... deploy@encircle360"
    ];
  };

  security.sudo.wheelNeedsPassword = false;
}

The K3s module encapsulates the entire Kubernetes host configuration:

# modules/k3s-node.nix
{ config, pkgs, ... }:

{
  # Firewall: K3s-relevant ports
  networking.firewall = {
    enable = true;
    allowedTCPPorts = [ 22 6443 10250 ];
    allowedUDPPorts = [ 8472 51820 ];
  };

  # Kernel parameters for container networking
  boot.kernel.sysctl = {
    "net.ipv4.ip_forward" = 1;
    "net.bridge.bridge-nf-call-iptables" = 1;
  };

  # Make kubectl available on the host
  environment.systemPackages = with pkgs; [
    kubectl
    k9s
  ];
}

The host-specific configuration then only contains what distinguishes this one server from the others -- hostname, IP address, K3s role:

# hosts/k3s-worker-01/configuration.nix
{ config, pkgs, ... }:

{
  imports = [ ./hardware-configuration.nix ];

  networking.hostName = "k3s-worker-01";
  networking.interfaces.ens3.ipv4.addresses = [{
    address = "10.0.1.11";
    prefixLength = 24;
  }];
  networking.defaultGateway = "10.0.1.1";

  services.k3s = {
    enable = true;
    role = "agent";
    serverAddr = "https://10.0.1.10:6443";
    tokenFile = "/etc/k3s/token";
  };

  system.stateVersion = "23.05";
}

This separation is crucial. When we change the SSH configuration, we modify base.nix once, and all nodes pick up the change on the next deployment. When we need to open a new K3s port, we modify k3s-node.nix. When a single node gets a different IP, we only change its host-specific file.

Remote Deployment: No More SSH-Copy

The second major improvement over our earlier workflow is remote deployment. Instead of copying the configuration to each server and running nixos-rebuild switch locally, we deploy from our local machine:

# Deploy a single node
nixos-rebuild switch --flake .#k3s-worker-01 \
  --target-host deploy@10.0.1.11 \
  --use-remote-sudo

# Deploy the server node
nixos-rebuild switch --flake .#k3s-server-01 \
  --target-host deploy@10.0.1.10 \
  --use-remote-sudo

The --flake .#k3s-worker-01 tells nixos-rebuild which configuration from flake.nix to use. --target-host specifies the target server. The build happens locally (or on a build server), the finished packages are copied to the target host via SSH and activated there.

This has several advantages. First, the target servers don't need a Git repository with the configuration. Second, we can test a build before deploying it -- nixos-rebuild build --flake .#k3s-worker-01 builds the configuration locally without changing anything on the server. Third, the entire deployment process is reproducible and controllable from a single location.

For updating all nodes, we have a simple script:

#!/usr/bin/env bash
set -euo pipefail

NODES=("k3s-server-01:10.0.1.10" "k3s-worker-01:10.0.1.11" "k3s-worker-02:10.0.1.12")

for entry in "${NODES[@]}"; do
  name="${entry%%:*}"
  host="${entry##*:}"
  echo "Deploying $name..."
  nixos-rebuild switch --flake ".#$name" \
    --target-host "deploy@$host" \
    --use-remote-sudo
  echo "$name deployed successfully."
done

No Ansible, no Puppet, no Terraform. The Nix configuration is simultaneously the infrastructure definition and the deployment tool. That means less tooling, less indirection, fewer sources of error.

Secrets: The Open Flank

Reproducible configuration has a natural tension point: secrets. The flake.nix and all modules reside in a Git repository. K3s tokens, SSH private keys, and other secrets don't belong there.

Our solution is pragmatic. Secrets are stored as files on the servers in /etc/secrets/ and are referenced through the Nix configuration but not defined in it. The K3s token, for example, is manually copied to the server during initial setup and only specified as a path in the configuration (tokenFile = "/etc/k3s/token").

This isn't elegant. Tools like agenix or sops-nix encrypt secrets so they can live in the repository and are only decrypted on the target server. We deliberately chose against this -- not because the tools are bad, but because our number of secrets is manageable and we don't currently need the additional complexity. With more nodes or more frequently rotating secrets, we would reconsider.

Lessons After Two Years

A few insights that don't come from the documentation:

Enable Flakes sooner. We worked for almost a year without Flakes. That was wasted time. Flakes solve so many everyday problems -- pinned versions, modularization, reproducible builds -- that we recommend them to anyone who wants to use NixOS seriously. With NixOS 23.05 and 23.11, Flakes are stable enough for any production use.

Don't forget Nix Store garbage collection. The Nix Store grows with every generation. Without regular garbage collection, old generations quickly consume double-digit gigabytes of disk space. nix.gc.automatic = true in the configuration is mandatory.

Modularize configuration from the start. We worked too long with monolithic configuration.nix files per node. The transition to modules was laborious but should have happened from day one. Anyone starting with more than one node should think in modules immediately.

nixos-rebuild build before switch. Sounds obvious, but it wasn't for us initially. Testing a build locally before pushing it to a production server is trivial with Flakes and saves a lot of stress.

Conclusion

Over two years, NixOS has evolved from a promising experiment to an indispensable part of our infrastructure. The combination of Nix Flakes for reproducible builds, shared modules for consistent configuration, and remote deployment for scalable management solves problems that previously required Ansible, manual scripts, and a lot of hope.

As we described in our K3s production article, we stick with the deliberate layer separation: NixOS as the host OS configures the machine, K3s runs on top as the Kubernetes distribution, and our applications run as containers in Kubernetes. Nix doesn't manage applications -- it manages the foundation on which everything else runs.

If you're evaluating NixOS for servers and have read our introductory article from 2022: Flakes make the decisive difference. The learning curve is still real, but the result is an infrastructure that we can call reproducible with a clear conscience. Not "should be reproducible." It is.