KubeVirt 1.0 in Practice: VMs and Containers United

From Evaluation to Production

In the summer of 2022, we described in our KubeVirt introductory article how KubeVirt treats virtual machines as native Kubernetes objects. Back then, the project was in pre-1.0 stage, and we concluded with the question of whether the limitations were acceptable for production scenarios. A year and a half later, we can answer that question: Yes -- with limitations you should be aware of.

In July 2023, KubeVirt reached version 1.0. This was more than a version number: it signaled API stability, defined deprecation cycles, and gave teams like ours the confidence that existing VM definitions wouldn't change with every minor upgrade. We're now on KubeVirt 1.2, the project remains CNCF Incubating, and the community continues to grow. Time for an honest field report.

Our Use Cases

We don't run KubeVirt out of technical curiosity -- we run it because it solves concrete problems. Three scenarios have established themselves for us.

Legacy applications. In client projects, we regularly encounter software that can't be containerized -- proprietary services with specific kernel requirements, applications with hardcoded file paths, or simply software whose vendor only delivers VM images. Instead of running a separate virtualization platform for these, these workloads now run as VirtualMachine objects on the same cluster as our containers.

Databases with performance requirements. Certain database workloads need predictable I/O latency and direct access to dedicated resources. In a VM with assigned CPU cores and hugepages, we can guarantee this better than in a container that shares resources with other pods.

Windows workloads. Yes, they still exist -- Windows services for which there's no Linux alternative. With KubeVirt, we can run Windows VMs on our Linux nodes, including VNC access via virtctl and automated provisioning through cloud-init-compatible Windows images.

Infrastructure: K3s on NixOS

As described in our articles on K3s in production and NixOS as a server operating system, we run our clusters on K3s nodes with NixOS as the host OS. For KubeVirt, we had to extend the NixOS configuration of the nodes -- specifically KVM support and device permissions:

# KubeVirt-related NixOS configuration
virtualisation.libvirtd.enable = true;
boot.kernelModules = [ "kvm-intel" "vhost_net" ];
boot.kernel.sysctl = {
  "net.bridge.bridge-nf-call-iptables" = 1;
  "net.ipv4.ip_forward" = 1;
};

The crucial point is that /dev/kvm must be available and accessible to the container runtime on all nodes. On NixOS, this is cleanly solvable through declarative configuration. NixOS's reproducibility plays to its strength here: when we add a new node to the cluster, KubeVirt capability is automatically included.

Installing KubeVirt itself has become simpler since 1.0. We use the operator in the current version:

# Install KubeVirt 1.2 Operator
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v1.2.0/kubevirt-operator.yaml

# Deploy KubeVirt CR
kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/v1.2.0/kubevirt-cr.yaml

# Install CDI for image import
kubectl apply -f https://github.com/cdi-project/containerized-data-importer/releases/download/v1.58.0/cdi-operator.yaml
kubectl apply -f https://github.com/cdi-project/containerized-data-importer/releases/download/v1.58.0/cdi-cr.yaml

VMs with Cloud-Init and Persistent Storage

In the introductory article, we showed a simple VM with containerDisk. In production, we work with persistent volumes and extensive cloud-init configuration. Here's an example that reflects our typical setup:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: legacy-app-01
  namespace: vms
  labels:
    app: legacy-app
    env: production
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: legacy-app-01
        app: legacy-app
    spec:
      domain:
        cpu:
          cores: 2
          model: host-passthrough
        memory:
          guest: "4Gi"
          hugepages:
            pageSize: "2Mi"
        resources:
          requests:
            memory: "4Gi"
          limits:
            memory: "4Gi"
        devices:
          disks:
            - name: rootdisk
              disk:
                bus: virtio
              bootOrder: 1
            - name: datadisk
              disk:
                bus: virtio
            - name: cloudinitdisk
              disk:
                bus: virtio
          interfaces:
            - name: default
              masquerade: {}
      networks:
        - name: default
          pod: {}
      volumes:
        - name: rootdisk
          dataVolume:
            name: legacy-app-01-root
        - name: datadisk
          persistentVolumeClaim:
            claimName: legacy-app-01-data
        - name: cloudinitdisk
          cloudInitNoCloud:
            userData: |
              #cloud-config
              hostname: legacy-app-01
              manage_etc_hosts: true
              users:
                - name: ops
                  sudo: ALL=(ALL) NOPASSWD:ALL
                  ssh_authorized_keys:
                    - ssh-ed25519 AAAA...
              packages:
                - qemu-guest-agent
                - prometheus-node-exporter
              runcmd:
                - systemctl enable --now qemu-guest-agent
                - systemctl enable --now prometheus-node-exporter
  dataVolumeTemplates:
    - metadata:
        name: legacy-app-01-root
      spec:
        source:
          http:
            url: "https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img"
        pvc:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 20Gi
          storageClassName: longhorn

A few details are worth highlighting. The dataVolumeTemplates section uses CDI to automatically download the Ubuntu cloud image and write it to a PVC -- when the VM is first created. On subsequent starts, the existing volume is reused. The host-passthrough CPU model gives the VM access to the host's actual CPU features, which is critical for performance. And the qemu-guest-agent in the cloud-init configuration allows KubeVirt to monitor the VM's state more accurately.

virtctl in Daily Use

virtctl is our daily tool for VM management. The most important commands we use regularly:

# VM lifecycle
virtctl start legacy-app-01 -n vms
virtctl stop legacy-app-01 -n vms
virtctl restart legacy-app-01 -n vms

# SSH access via the Kubernetes API server (no NodePort needed)
virtctl ssh ops@legacy-app-01 -n vms

# Console for debugging without network
virtctl console legacy-app-01 -n vms

# Trigger live migration manually
virtctl migrate legacy-app-01 -n vms

# Create VM snapshot (stable since KubeVirt 1.0)
virtctl snapshot create legacy-app-01 --name=pre-upgrade-snapshot -n vms

# Port forwarding for local access
virtctl port-forward legacy-app-01 8080:80 -n vms

virtctl ssh in particular has changed our workflow. Instead of exposing SSH via NodePorts or LoadBalancers, virtctl tunnels the connection through the Kubernetes API. This significantly simplifies network configuration and reduces the attack surface. The snapshot functionality, which became stable with 1.0, we use consistently before VM upgrades -- a safety net that doesn't exist with traditional containers.

Live Migration: Disillusionment and Success

Live migration was one of the features that impressed us most in the introductory article. In practice, we've experienced both successes and limitations.

What works reliably: VMs with moderate memory consumption (up to about 8 GB) migrate between nodes in a few seconds. This is crucial for node maintenance -- a kubectl drain on a node automatically triggers the migration of VMs, and the downtime window stays in the millisecond range.

Where it gets more complicated: VMs with large memory footprints and high write rates. A database VM with 16 GB of RAM and active write operations takes significantly longer because KubeVirt copies memory incrementally while the VM continues writing. In one case, we had to briefly pause the VM to complete the migration. For such workloads, we deliberately plan maintenance windows.

The prerequisite for live migration remains shared storage. On our nodes, we use Longhorn, which provides the necessary flexibility with ReadWriteMany volumes.

Networking: Multus for Secondary Networks

Standard networking via pod networks and masquerade interfaces is sufficient for many scenarios. But in client projects, VMs frequently need access to VLANs or dedicated network segments -- for example, legacy applications that need to be reachable by IP address from a specific subnet.

For this, we use Multus CNI, which enables multiple network interfaces per pod (and thus per VM). A VM can thus be connected to the pod network and simultaneously receive a second interface into a VLAN:

spec:
  template:
    spec:
      domain:
        devices:
          interfaces:
            - name: default
              masquerade: {}
            - name: vlan100
              bridge: {}
      networks:
        - name: default
          pod: {}
        - name: vlan100
          multus:
            networkName: vlan100-net-attach

The corresponding NetworkAttachmentDefinition configures the VLAN interface on the nodes. This requires preparatory work on the NixOS side -- the physical interfaces must be available for Multus -- but once set up, it works reliably.

Monitoring with Prometheus

KubeVirt exposes comprehensive metrics via a Prometheus-compatible endpoint. This allows VM monitoring to integrate seamlessly into our existing Prometheus stack. The key metrics we monitor:

• kubevirt_vmi_memory_resident_bytes -- actual memory consumption of the VM
• kubevirt_vmi_vcpu_seconds_total -- CPU usage per vCPU
• kubevirt_vmi_storage_read_traffic_bytes_total -- disk I/O
• kubevirt_vmi_network_receive_bytes_total -- network traffic
• kubevirt_vmi_migration_data_processed_bytes -- progress during live migrations

Additionally, we install the prometheus-node-exporter into every VM via cloud-init. This gives us the same host-level metrics we know from our physical nodes -- filesystem utilization, systemd service status, detailed network statistics. The combination of KubeVirt metrics (hypervisor level) and node exporter metrics (guest level) gives us a complete picture.

Our Grafana dashboards display VMs and containers side by side. An operator can see at a glance whether a VM is alive, how its resources are utilized, and whether a migration is running. This was one of our goals: a unified operational view across both worlds.

What Has Improved Since 1.0

The concerns we expressed in the introductory article can largely be dispelled. Documentation has improved massively -- the KubeVirt User Guide now covers more complex scenarios as well. API stability since 1.0 gives us planning certainty for upgrades. And the snapshot and restore functionality, which was still experimental back then, we now use regularly in production.

What's still missing is a mature GUI. There are approaches -- the KubeVirt plugin for the Rancher UI and the OpenShift console -- but a standalone, lightweight VM manager comparable to Proxmox or vCenter doesn't exist. For us as a CLI-oriented team, this isn't a problem. For teams that expect graphical management, it remains a hurdle.

Conclusion

KubeVirt has made the leap from promising technology to production-ready solution. The 1.0 was the turning point -- not because the technology fundamentally changed, but because it was the signal for API stability and long-term support. On our K3s clusters with NixOS, we now run a good dozen VMs alongside our container workloads, and the two worlds no longer feel like separate systems.

The biggest win is operational: one team, one platform, one monitoring stack, one deployment workflow. If you're running Kubernetes today and have VM workloads -- whether legacy software, Windows services, or performance-sensitive databases -- you should seriously evaluate KubeVirt. The era of "interesting, but not yet mature" is over.