KubeVirt: Running Virtual Machines on Kubernetes

When Containers Are Not Enough

At encircle360, we have been relying on Kubernetes and containers for years. Our entire stack -- from Gitea to Keycloak to Directus -- runs containerized on K3s nodes with NixOS. For our own services, this works excellently. But in client projects, we regularly hit a limit: not every workload can be meaningfully packed into a container.

Legacy applications that have been running on a dedicated VM for ten years and whose internals nobody fully understands anymore. Windows services for which there simply is no container image. Databases that expect bare-metal-level performance and direct disk access. Or simply regulatory requirements that mandate a specific operating system version.

Until now, this meant two separate worlds. Containers on Kubernetes, VMs on a separate hypervisor. Two platforms, two toolchains, two operational concepts. KubeVirt promises to bridge this divide.

What Is KubeVirt?

KubeVirt is a Kubernetes add-on that extends the Kubernetes API with the ability to create and manage virtual machines. The project has been a CNCF Incubating project since 2019 and is actively developed -- the current version at the time of this article is approximately 0.55. Red Hat and a growing community are among the key contributors behind the project.

The central idea: VMs become Kubernetes-native objects. You define them as YAML, deploy them with kubectl apply, monitor them with the familiar Kubernetes tools, and manage their lifecycle through the Kubernetes API. Under the hood, KubeVirt uses KVM and QEMU -- proven Linux virtualization technology. The VM runs inside a pod that contains a QEMU process. From the outside, Kubernetes sees a pod. Inside that pod runs a full virtual machine with its own kernel.

CRDs: VirtualMachine and VirtualMachineInstance

KubeVirt introduces several Custom Resource Definitions, two of which are central.

VirtualMachineInstance (VMI) represents a running VM instance. Comparable to a pod: when the VMI is deleted, the VM is gone. For short-lived or stateless VMs, this may be sufficient.

VirtualMachine (VM) is the persistent abstraction on top. It defines the desired VM configuration and manages the lifecycle of the underlying VMI. A VirtualMachine can be stopped and restarted without losing its definition -- analogous to a Deployment that manages pods.

A simple example of a VirtualMachine definition:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: test-vm
  namespace: kubevirt-demo
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: test-vm
    spec:
      domain:
        resources:
          requests:
            memory: "1Gi"
        devices:
          disks:
            - name: rootdisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
      volumes:
        - name: rootdisk
          containerDisk:
            image: quay.io/kubevirt/fedora-cloud-container-disk-demo:latest
        - name: cloudinitdisk
          cloudInitNoCloud:
            userData: |
              #cloud-config
              hostname: test-vm
              ssh_authorized_keys:
                - ssh-rsa AAAA...
              packages:
                - htop
                - curl

Anyone accustomed to YAML definitions for Kubernetes will feel right at home. The VM specification defines resources, disks, and volumes -- except here it is not about container images but about VM disk images and cloud-init configuration. The containerDisk volume is a clever trick: the VM image is packaged as a container image and distributed via a container registry. This leverages the existing Kubernetes infrastructure for image distribution.

virtctl: The VM Command Line

While kubectl suffices for most operations, KubeVirt provides virtctl as a complementary CLI tool for VM-specific actions.

# VM starten und stoppen
virtctl start test-vm
virtctl stop test-vm

# Konsole öffnen (wie SSH, aber direkt)
virtctl console test-vm

# VNC-Verbindung für grafische Oberfläche
virtctl vnc test-vm

# VM-Image als Volume exponieren
virtctl expose vmi test-vm --port=22 --name=test-vm-ssh --type=NodePort

virtctl console in particular impressed us. Direct console access to the VM, without network configuration, without SSH setup -- simply through the Kubernetes API. For debugging and initial setup, this is extremely practical.

Persistent Volumes for VM Disks

Containers are typically ephemeral -- everything is gone on restart. For VMs, this is obviously not acceptable. KubeVirt uses Kubernetes PersistentVolumeClaims to permanently store VM disks. This means: any storage provider that the Kubernetes cluster supports also works as VM storage.

On our K3s nodes with NixOS, we use Longhorn as distributed storage. This allows VM disks to be replicated across nodes -- a prerequisite for live migration, which we will discuss shortly.

volumes:
  - name: rootdisk
    persistentVolumeClaim:
      claimName: fedora-vm-disk

The challenge lies less in configuration than in performance. VM workloads place different demands on storage than containers. Databases in VMs in particular need low latency and high throughput. Whether network-based storage like Longhorn is sufficient for this depends heavily on the specific workload. For our tests, performance was acceptable, but we would evaluate this more closely on a case-by-case basis for production database VMs.

CDI: Importing VM Images

Where do the VM images come from? In addition to the previously mentioned container disks, the Containerized Data Importer (CDI) offers an elegant solution. CDI is a separate operator that imports VM images from various sources and provides them as PersistentVolumes.

Supported sources include HTTP URLs, S3-compatible storage, and existing disk images in formats like QCOW2 or RAW. In practice, this means: you can take existing VM images from VMware, OpenStack, or a local libvirt environment without having to manually repackage them into container images.

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  name: fedora-dv
spec:
  source:
    http:
      url: "https://download.fedoraproject.org/pub/fedora/linux/releases/36/Cloud/x86_64/images/Fedora-Cloud-Base-36-1.5.x86_64.qcow2"
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi

CDI downloads the image, converts it if necessary, and writes it to a PVC. After that, the VirtualMachine can reference this volume. The entire process runs as a Kubernetes Job -- fully automated and observable.

Live Migration

One of the features that impressed us most is live migration. KubeVirt can move running VMs from one node to another -- without downtime. The prerequisites are shared storage (for example via Longhorn or Ceph) and a compatible CPU architecture on the target nodes.

# Live Migration auslösen
virtctl migrate test-vm

For Kubernetes operations like node drains during updates, this is crucial. Without live migration, a VM would have to be shut down and restarted on another node. With live migration, the memory state is incrementally copied to the target node while the VM continues running. The actual outage is limited to a few milliseconds during the final switchover.

On our K3s nodes, we tested this successfully: kubectl drain on a node automatically triggers migration of the VMs to other nodes. The behavior is equivalent to what you know from VMware vMotion or OpenStack live migration -- just integrated into the Kubernetes platform.

KubeVirt on K3s with NixOS

As described in our post on K3s in production, we run our clusters on K3s nodes with NixOS as the host operating system. Installing KubeVirt on K3s requires a few adjustments.

The nodes must support hardware virtualization -- KVM must be available. On NixOS, you enable this through the appropriate kernel configuration. Additionally, the /dev/kvm device must be accessible in the containerd configurations. We had to extend the NixOS configuration of our nodes by a few lines, but the effort was manageable.

The installation of KubeVirt itself is done via the KubeVirt operator:

# KubeVirt Operator installieren
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v0.55.0/kubevirt-operator.yaml

# KubeVirt Custom Resource erstellen
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v0.55.0/kubevirt-cr.yaml

# Status prüfen
kubectl -n kubevirt wait kv kubevirt --for condition=Available --timeout=300s

After a few minutes, KubeVirt is ready to use. The custom resources are registered, the necessary DaemonSets are running on all nodes, and virtctl can control VMs.

Where Do We See the Use Cases?

After several weeks of experimentation, we see KubeVirt not as a replacement for dedicated virtualization platforms but as a bridge. Its strength lies in scenarios where containers and VMs need to coexist on the same platform.

Legacy migration: Workloads that cannot be containerized immediately can run as VMs on Kubernetes and be modernized step by step. This avoids the big-bang approach and allows for a gradual migration.

Unified operations: One team, one platform, one set of tools. Instead of running Kubernetes and VMware in parallel, you consolidate on a single platform. Monitoring, networking, storage, and RBAC work uniformly.

Development environments: Complete system landscapes -- including VMs for components that are not containerized -- can be defined as Kubernetes manifests and set up reproducibly.

Open Questions

KubeVirt is in active pre-1.0 development, and this shows in some areas. The documentation has gaps, particularly for more complex network configurations. Performance tuning for I/O-intensive workloads requires deep understanding of QEMU and KVM. And the question of how well KubeVirt scales with hundreds of VMs is one we cannot answer with our small setups.

The tooling ecosystem is also still young. There is no mature GUI-based VM manager comparable to vCenter. Those who want to manage VMs via YAML files and CLI will feel at home. Those who need a graphical console for management will have to wait a bit longer.

Conclusion

KubeVirt solves a problem that the container world likes to ignore: not everything can and should be a container. Instead of ignoring this reality, KubeVirt extends Kubernetes with the missing capability -- and does so in a way that feels native and well-thought-out. VMs as Kubernetes objects, managed with the same tools, on the same infrastructure.

For our client projects, we see concrete potential here. The coming months will show how KubeVirt evolves on its path to version 1.0 and whether the pre-1.0 limitations are acceptable for production scenarios. At the moment, we are impressed by the concept and will follow the project closely. Anyone running Kubernetes who has VM workloads should have KubeVirt on their radar.