CI Runners in a microVM: Docker Builds with Kata Containers on Kubernetes

Why CI jobs should not run on the host kernel

From a security standpoint, a CI runner is an awkward piece of machinery: it executes code that changes fast, comes from many hands and pulls half a registry through its dependencies — and for Docker builds it needs a Docker daemon whose container chart hardcodes privileged: true. On a classic Kubernetes node, that job shares the kernel with everything else running there. One kernel exploit in a build step, and the line between CI job and cluster node is gone.

We had previously mitigated this with Sysbox: dedicated user namespaces, virtualized /proc and /sys, dockerd effectively unprivileged at the host level. That is far better than raw privileged — but the kernel stays shared. Namespace hardening shifts the attack surface; it does not remove it.

Kata Containers takes the other route: every pod gets its own lightweight virtual machine with its own guest kernel. A container escape then ends at KVM's hardware virtualization boundary instead of the host kernel. That is exactly what we wanted for our Gitea Actions runners.

The integration: one RuntimeClass, no new deployment model

The beauty of Kata is how unspectacular the Kubernetes side is. Kata registers as an additional containerd runtime next to runc, and the cluster gets a RuntimeClass for it:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata
overhead:
  podFixed:
    memory: "160Mi"
    cpu: "250m"
scheduling:
  nodeSelector:
    example.com/kata: "enabled"

Workloads opt in with a single line: runtimeClassName: kata in the pod spec. The overhead field makes the scheduler account for the VM cost per pod, and the nodeSelector keeps pods off nodes where Kata is not installed — we set that label together with the runtime installation, both coming declaratively from the same NixOS configuration and flowing through the same GitOps workflow as everything else in our clusters.

On the containerd side, one detail matters if you plan privileged workloads like Docker-in-Docker (see the CRI plugin configuration):

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata]
  runtime_type = "io.containerd.kata.v2"
  privileged_without_host_devices = true

privileged_without_host_devices ensures a privileged container gets full rights inside its VM while the host's device files stay outside — the Kata documentation recommends exactly this setting for all Kata runtimes. The dind sidecar's privileged: true now only applies to the guest VM — exactly the semantics you want.

Docker-in-Docker inside the VM: the five pitfalls

So much for theory. In practice our runner pod started under Kata right away — but the Docker daemon inside took five attempts before the first build went through. The order below is the order in which the problems surfaced. If you move CI runners to Kata, you will likely meet every single one of them.

1. overlay2 does not work on virtiofs. Kata serves the pod filesystem and filesystem volumes into the VM via virtiofs. Docker's default overlay2 storage driver cannot stack an overlay on top of it — dockerd dies at startup with failed to mount overlay: invalid argument. The fix is fuse-overlayfs as the storage driver. The catch: the official docker:dind image does not ship the binary (only the rootless variant does). We build a minimally derived image — a two-line Dockerfile that adds fuse-overlayfs.

2. /dev/fuse does not exist in the container. The Kata guest kernel has FUSE built in (virtiofs itself relies on it), but Kata deliberately does not populate guest device nodes into containers — the flip side of privileged_without_host_devices. fuse-overlayfs aborts with fuse: device not found. Since the container is privileged inside the VM, it holds CAP_MKNOD: a mknod /dev/fuse c 10 229 before the entrypoint fixes it for good.

3. Extended attributes need a virtiofsd flag. The first image pull failed mid-extraction: lsetxattr ... security.capability: operation not supported. Images containing files with file capabilities — which includes many CI base images — need xattr support along the storage path. virtiofsd disables it by default for performance; adding --xattr to the virtio_fs_extra_args in the Kata configuration resolves it.

4. The MTU trap. A classic in new clothing: if pod networking rides an overlay with a small MTU (1280 in our case), the VM inherits that MTU on its interface — but the Docker bridge inside the VM defaults to 1500. Build containers can open TCP connections, yet large packets vanish into a black hole: apk fetch starts and hangs. A dockerd --mtu=1240 ends the haunting.

5. Unix sockets cannot be shared between pod containers. This was the hardest lesson, because three mechanisms fail in sequence: virtiofs cannot carry unix sockets. EmptyDir volumes — even with medium: Memory — are materialized by Kata as a separate instance per mount, so they share nothing between containers. And subPath mounts are resolved by the kubelet on the host, where the guest content does not exist. The classic trick of handing dockerd's socket to the runner container via a volume is simply impossible under Kata.

Our final architecture therefore separates two paths. The runner process talks to dockerd over TLS on pod-local TCP (port 2376 — the dind entrypoint generates the certificates itself if you leave its TLS default mode enabled). The certificates are ordinary files and live on a small shared PVC — the only volume type that is actually shared between pod containers under Kata. The job containers, in contrast, get the real unix socket via a dockerd-side bind mount: dockerd resolves the source path in its own namespace, and the mount is a kernel bind entirely inside the VM. No unencrypted TCP listener, no socket contortions, and Docker's already-deprecated plaintext API port stays off.

The security balance sheet

In the end, the honest comparison with the starting point is what counts. Versus Sysbox we gain the hardware boundary: a kernel exploit from a build job compromises the guest kernel of a disposable VM, not the node. The dind container's hardcoded privileged is confined to the VM, host devices are locked out, the ServiceAccount token stays unmounted, and a default-deny NetworkPolicy additionally seals the runner namespace — a CI runner has no business receiving inbound traffic anyway.

Equally honest: the relevant attack surface shifts to QEMU and virtiofsd, which run as host processes per pod. That is why we care that both come from the package manager and are refreshed with every regular system update — not as frozen binaries aging in /opt for years. And the --xattr flag from pitfall 3 slightly enlarges virtiofsd's parsing surface; we accept that consciously.

Operations and updates

Day to day, the difference from running on runc is smaller than expected. The guest kernel is updated with the Kata package and flows through the same update channel as everything else; running pods keep their VM until the next pod restart, and no host reboot is required. The rule of thumb we wrote down: after every Kata update, roll the long-lived Kata pods once. The resource overhead is around 160 MiB per pod plus a few seconds of VM startup — for CI runners executing minutes-long jobs, both are irrelevant.

Lessons learned

The Kubernetes side is trivial, the Docker side is not. Creating the RuntimeClass and adding one line to the pod spec was an afternoon. The five pitfalls above cost a multiple of that, because each one only became visible after the previous one was solved. If you are planning the move, read that list as a checklist, not as an anecdote.

Volume semantics under Kata are a discipline of their own. The assumption that pod containers share an emptyDir runs deep — under Kata it is wrong. When migrating existing multi-container pods, think through every volume interaction between containers individually: files over PVCs work, sockets do not, and subPath is off limits.

A real build is the only valid test. Our smoke test — build an image, run the container, rebuild from cache — only went green after all five problems were solved. After that, the first real pipeline (a Keycloak image build with a push to our internal registry) passed on the first attempt. kata-runtime check only tells you the VM can start; whether your workload can live inside it, only the workload can tell.

Conclusion

Kata Containers delivers on its promise: CI jobs with a privileged Docker daemon now run behind a hardware boundary instead of on the shared kernel, and in everyday Kubernetes life it feels exactly like before — one RuntimeClass, nothing else. The price is neither performance nor operational overhead, but a handful of very specific integration problems around Docker-in-Docker that you have to solve and document once. That is precisely what this article is for.