K3s and KubeVirt: Converged Infrastructure on Bare Metal

Two Worlds, One Platform

Over the past few years, we at encircle360 have been running two infrastructure tracks in parallel. On one side, our K3s clusters for container workloads, as we described in our K3s production report. On the other side, dedicated hosts for virtual machines -- whether via libvirt, Proxmox, or in some cases simply QEMU on the command line. KubeVirt had already begun to blur this separation, as we described in our articles on KubeVirt and KubeVirt 1.0 in practice. But the hosts themselves remained separate: VPS instances for Kubernetes, dedicated servers for VMs.

Since fall 2024, we have taken this a step further. We run K3s and KubeVirt on bare-metal servers with NixOS -- a converged infrastructure that unifies containers and VMs on the same physical machines. No more separate hypervisor cluster, no second set of hosts for VM workloads. One platform for everything.

Why Bare Metal?

The answer is simple: KubeVirt needs KVM, and KVM needs hardware virtualization. On VPS instances from common hosting providers, nested virtualization is either unavailable or so slow that it is not viable for production VMs. Anyone who wants to use KubeVirt seriously needs bare metal with VT-x or AMD-V enabled.

Our current nodes are dedicated servers with 64 GB RAM each, 8-core CPUs, and NVMe storage. That sounds like a lot of hardware for a team our size, but that is exactly the point: these machines replace both the previous K3s VPS instances and the separate VM hosts. Consolidating onto fewer, more powerful machines reduces overall costs and operational overhead.

Hardware requirements at a glance:

• CPU: VT-x/AMD-V mandatory, ideally with VT-d for device passthrough
• RAM: At least 32 GB per node, preferably 64 GB -- VMs and containers share the memory
• Storage: NVMe recommended for VM disk I/O, additional SSDs for Longhorn replication
• Network: At least 1 Gbit, ideally 10 Gbit for live migration between nodes

NixOS as the Foundation

Our bare-metal hosts run NixOS, which we have been using as our host operating system since 2022. In our NixOS introduction article and the Flakes follow-up article, we described the basics. For the converged infrastructure, we needed to extend the NixOS configuration with KVM, libvirt, and the necessary kernel modules.

The KubeVirt module in our Flake-based configuration looks like this:

# modules/kubevirt-host.nix
{ config, pkgs, ... }:

{
  # Enable KVM and virtualization
  virtualisation.libvirtd.enable = true;
  virtualisation.libvirtd.qemu = {
    package = pkgs.qemu_kvm;
    ovmf.enable = true;
  };

  # Kernel modules for KVM and networking
  boot.kernelModules = [
    "kvm-intel"    # or kvm-amd depending on CPU
    "vhost_net"
    "br_netfilter"
    "tun"
  ];

  # Make /dev/kvm accessible to container runtime
  services.udev.extraRules = ''
    KERNEL=="kvm", GROUP="kvm", MODE="0666"
  '';

  # Kernel parameters for bridge networking and IP forwarding
  boot.kernel.sysctl = {
    "net.bridge.bridge-nf-call-iptables" = 1;
    "net.bridge.bridge-nf-call-ip6tables" = 1;
    "net.ipv4.ip_forward" = 1;
  };

  # Hugepages for VM performance (optional but recommended)
  boot.kernelParams = [
    "hugepagesz=2M"
    "hugepages=4096"
  ];

  # Additional packages on the host
  environment.systemPackages = with pkgs; [
    qemu
    libvirt
    virtctl
    bridge-utils
  ];
}

The key points: virtualisation.libvirtd.enable activates the libvirt daemon and ensures KVM is loaded correctly. The udev rule ensures that /dev/kvm is accessible to all processes -- without this, KubeVirt cannot start VMs. The hugepages configuration reserves memory for VMs with predictable performance. In our Flake, we reference this module alongside the existing k3s-node.nix:

# Excerpt from flake.nix
k3s-bare-01 = nixpkgs.lib.nixosSystem {
  system = "x86_64-linux";
  modules = [
    ./hosts/k3s-bare-01/configuration.nix
    ./modules/base.nix
    ./modules/k3s-node.nix
    ./modules/kubevirt-host.nix
  ];
};

A nixos-rebuild switch on the bare-metal nodes activates KVM, loads the kernel modules, and makes the machine ready for KubeVirt -- reproducibly and with rollback capability. When we add a new node, it is automatically KubeVirt-ready after deploying the Flake configuration.

K3s with KubeVirt Operator

K3s runs on the bare-metal nodes as usual as a NixOS service. For the KubeVirt installation, we switched from individual kubectl apply commands to Helm. This gives us better control over configuration and upgrades:

# Install KubeVirt Operator via Helm
helm repo add kubevirt https://kubevirt.github.io/kubevirt-chart
helm repo update

helm install kubevirt kubevirt/kubevirt \
  --namespace kubevirt \
  --create-namespace \
  --version 1.4.0 \
  --set operator.image.tag=v1.4.0

# CDI for image import
helm repo add cdi https://kubevirt.github.io/containerized-data-importer-chart
helm install cdi cdi/cdi \
  --namespace cdi \
  --create-namespace \
  --version 1.60.0

The Helm installation allows us to version KubeVirt configuration as a values file and roll it out through our existing Helmfile pipeline. After installation, we verify that all components are running:

# Check KubeVirt status
kubectl get kubevirt -n kubevirt
kubectl get pods -n kubevirt

# Verify KVM capability of nodes
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}: {.status.allocatable.devices\.kubevirt\.io/kvm}{"\n"}{end}'

Each node should report devices.kubevirt.io/kvm: "1k" as an allocatable resource. If that is missing, something is wrong with the KVM configuration on the host -- typically a missing kernel module or incorrect permissions on /dev/kvm.

MetalLB: LoadBalancer IPs on Bare Metal

On VPS instances with cloud providers, there is typically an external LoadBalancer service. On bare metal, this integration does not exist. Kubernetes services of type LoadBalancer remain stuck in Pending status because no controller assigns an external IP.

MetalLB solves this problem. It implements the LoadBalancer service type for bare-metal environments by assigning IP addresses from a configurable pool and announcing them via ARP or BGP on the network:

# Install MetalLB
helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb \
  --namespace metallb-system \
  --create-namespace

After installation, we define an IP pool and the announcement mode:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: default-pool
  namespace: metallb-system
spec:
  addresses:
    - 10.0.1.200-10.0.1.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default
  namespace: metallb-system
spec:
  ipAddressPools:
    - default-pool

With this, Kubernetes services of type LoadBalancer automatically receive an IP from the pool. This applies equally to container services and KubeVirt VMs exposed through a service. Traefik as the Ingress Controller gets its external IP this way, and SSH access to VMs can also be elegantly handled through LoadBalancer services.

Storage: Longhorn for the Converged Platform

For storage, we use Longhorn, which has already proven itself on our K3s clusters. On bare metal, Longhorn particularly shines: the NVMe disks of the nodes are used directly, without the I/O virtualization layer of a hosting provider in between.

Longhorn replicates volumes across nodes -- a prerequisite for live migration of KubeVirt VMs. When a VM migrates from Node A to Node B, its storage must already be available on Node B. With Longhorn and a replication factor of 2, this is automatically guaranteed.

For workloads that need maximum I/O performance -- such as database VMs -- we alternatively use the local-path-provisioner that comes with K3s. Local-Path writes directly to the local NVMe disk without replication. This is faster, but the VM is bound to the node and cannot be live migrated. We make the choice between Longhorn and Local-Path on a per-workload basis.

A VM on the Converged Platform

All together, this results in a platform where container deployments and VMs coexist side by side. Here is an example of a VM running on our bare-metal infrastructure:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: build-agent-01
  namespace: vms
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: build-agent-01
        role: build-agent
    spec:
      domain:
        cpu:
          cores: 4
          model: host-passthrough
        memory:
          guest: "8Gi"
          hugepages:
            pageSize: "2Mi"
        devices:
          disks:
            - name: rootdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
            - name: default
              masquerade: {}
      networks:
        - name: default
          pod: {}
      volumes:
        - name: rootdisk
          dataVolume:
            name: build-agent-01-root
        - name: cloudinitdisk
          cloudInitNoCloud:
            userData: |
              #cloud-config
              hostname: build-agent-01
              users:
                - name: ops
                  sudo: ALL=(ALL) NOPASSWD:ALL
                  ssh_authorized_keys:
                    - ssh-ed25519 AAAA...
              packages:
                - qemu-guest-agent
                - docker.io
                - git
              runcmd:
                - systemctl enable --now qemu-guest-agent
                - systemctl enable --now docker
  dataVolumeTemplates:
    - metadata:
        name: build-agent-01-root
      spec:
        source:
          http:
            url: "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
        pvc:
          accessModes:
            - ReadWriteMany
          resources:
            requests:
              storage: 50Gi
          storageClassName: longhorn

The VM uses host-passthrough for CPU, hugepages for predictable memory, Longhorn for replicated storage, and Cloud-Init for automatic provisioning. It runs alongside our container workloads on the same node, shares the same pod network, and is monitored with the same tools.

Why This Replaces Separate Hosts

Consolidating onto a single platform brings advantages that go beyond mere cost savings:

One operational model. Instead of running Kubernetes for containers and Proxmox or libvirt for VMs, there is one platform with one API, one monitoring stack, and one deployment workflow. This reduces training effort and cognitive load on the team.

Flexible resource utilization. On separate hosts, resources are statically allocated. A VM host with 64 GB RAM where only 20 GB is used for VMs wastes 44 GB. On the converged platform, containers use the free memory. Utilization increases, cost per workload decreases.

Unified networking. VMs and containers share the Kubernetes network. Services can communicate directly with each other without NAT hacks or VPN tunnels between separate platforms. MetalLB assigns LoadBalancer IPs to both worlds equally.

Declarative from top to bottom. NixOS configures the host declaratively. Kubernetes manifests define containers and VMs declaratively. Helm and Helmfile orchestrate the deployments. The entire stack is versioned and reproducible -- from the kernel module to the VM's Cloud-Init script.

Limitations and Trade-offs

A converged platform is not a silver bullet. The complexity of individual nodes increases -- a bare-metal node with K3s, KubeVirt, Longhorn, and MetalLB has more moving parts than a simple K3s VPS. When a node fails, both containers and VMs are affected, not just one of the two worlds.

KubeVirt on K3s is also a combination that is less broadly tested than KubeVirt on a full Kubernetes distribution with OpenShift or Rancher. We have encountered occasional edge cases -- such as configuring device plugins for KVM on K3s, where documentation is thin. But nothing that could not be resolved.

The decision between Longhorn and Local-Path per workload requires discipline. Putting everything on Longhorn gives you live migration but comes with I/O overhead. Putting everything on Local-Path gives you performance but loses flexibility. Finding the right balance is an ongoing task.

Conclusion

K3s and KubeVirt on bare-metal NixOS hosts create a converged infrastructure that is just right for a team our size. No more separate VM hosts, no second management interface, no duplicate monitoring. Containers and VMs run side by side on the same machines, managed with the same tools, monitored with the same stack.

The path to get there was not trivial. NixOS needs to configure KVM correctly, MetalLB needs a clean IP plan, Longhorn needs to be sized for VM storage requirements. But the individual pieces are all mature -- K3s is a proven Kubernetes distribution, KubeVirt is past its 1.0, NixOS provides the declarative foundation. The art lies in the integration, and the investment has been worth it.

Anyone running bare-metal servers who needs both containers and VMs should consider this approach. The alternative -- running two separate platforms in parallel -- costs more money and more headaches in the long run than a one-time consolidation onto a converged platform.