CKS: clusters never become Ready on default KVM guest CPU model — Calico binaries (3.32.2) require x86-64-v2, failure is undiscoverable

[kiranchavala]

problem

A CKS cluster deployed on KVM hosts using CloudStack's default guest CPU configuration comes up with
every node NotReady and no CNI. The cause is that the CNI's binaries refuse to execute on the CPU
CloudStack exposes to the guest, but nothing in CloudStack surfaces this — the only evidence is in the log
of an init container three levels down.

versions

• ACS: 4.23
• Hypervisor: KVM on Oracle Linux 8 , Ubuntu 24
• Kubernetes: v1.37.0
• CNI: Calico v3.32.2 (quay.io/calico/cni:v3.32.2, quay.io/calico/node:v3.32.2)
• Guest CPU: guest.cpu.mode unset in /etc/cloudstack/agent/agent.properties (CloudStack default)

The steps to reproduce the bug

1. Build a CKS binaries ISO with Calico as the CNI**

./create-kubernetes-binaries-iso.sh /var/www/html 1.37.0 1.9.1 1.37.0 \
  https://raw.githubusercontent.com/projectcalico/calico/v3.32.2/manifests/calico.yaml \
  0.45.0 setup-v1.37.0 amd64 3.7.0

2. Register it and create a cluster

cmk add kubernetessupportedversion name=cks-1.37.0 semanticversion=1.37.0 \
  url=http://<host>/setup-v1.37.0-x86_64.iso zoneid=<zone> mincpunumber=2 minmemory=2048

cmk create kubernetescluster name=cks-test zoneid=<zone> \
  kubernetesversionid=<id> serviceofferingid=<so> size=1

3. Confirm the guest CPU model is the default

On a KVM host, before deploying:

grep -n "^guest.cpu" /etc/cloudstack/agent/agent.properties    # expect: no output

Observed on the cluster nodes:

$ kubectl get nodes
NAME                            STATUS     ROLES           AGE     VERSION
test-cks-control-1a0a9b3c3cb    NotReady   control-plane   5m38s   v1.37.0
test-cks-node-1a0a9b420dc       NotReady   <none>          5m27s   v1.37.0

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS       RESTARTS   AGE
kube-system   calico-kube-controllers-db57f7644-m6kln   0/1     Pending      0          6m49s
kube-system   calico-node-s8xs2                         0/1     Init:Error   6          6m45s
kube-system   calico-node-sqxp9                         0/1     Init:Error   6          6m49s
kube-system   coredns-559f6c778d-v8l2m                  0/1     Pending      0          6m50s
kube-system   coredns-559f6c778d-wdr4l                  0/1     Pending      0          6m50s
kube-system   headlamp-65978d54db-psb98                 0/1     Pending      0          6m48s
kube-system   kube-apiserver-test-cks-control-...       1/1     Running      0          6m56s
kube-system   kube-controller-manager-...               1/1     Running      0          6m56s
kube-system   kube-proxy-srbqg                          1/1     Running      0          6m45s
kube-system   kube-scheduler-...                        1/1     Running      0          6m56s

The control plane is healthy; only the CNI fails. Everything Pending is downstream of having no pod
network.

The actual error is in the first init container of calico-node:

$ kubectl -n kube-system logs calico-node-sqxp9 -c upgrade-ipam --previous
This program can only be run on AMD64 processors with v2 microarchitecture support.

That message comes from the Go runtime's startup CPU check, emitted when a binary is compiled with
GOAMD64=v2. Calico v3.32's binaries are built for the x86-64-v2 microarchitecture level, which
requires popcnt, sse3, ssse3, sse4_1, sse4_2, cmpxchg16b and lahf_lm.

CloudStack's KVM agent leaves the guest CPU model unset by default (guest.cpu.mode in
agent.properties, read at LibvirtComputingResource.java:1441-1453), so libvirt gives guests the generic
qemu64 model, which is x86-64-v1 only. The binary starts, checks CPUID, and exits 1.

Workaround

Confirm the cause by fixing the CPU model

On each KVM host:

echo "guest.cpu.mode=host-model" >> /etc/cloudstack/agent/agent.properties
systemctl restart cloudstack-agent

Recreate the cluster (the CPU model is applied at VM start, so existing nodes keep the old one). The nodes
now expose the v2 feature set and Calico starts normally:

What to do about it?

The workaround is guest.cpu.mode=host-model (or host-passthrough) on every KVM host, but CloudStack
gives operators no way to discover that from the symptom. Suggestions, roughly in order of value:

1. Document the requirement for CKS. The CKS documentation should state that KVM hosts need a guest
   CPU model exposing x86-64-v2 (i.e. guest.cpu.mode=host-model or host-passthrough), because current
   CNI images require it. This alone would have saved the whole investigation.

2. Surface the guest CPU model in CloudStack. guest.cpu.mode being per-host agent.properties with
   no global/cluster setting and no UI surface is the core usability problem: an admin cannot see or change
   what CPU their guests get without SSH-ing to every host. Exposing it as a host detail (alongside
   Host.OS.Kernel.Version etc., which are already reported) would make mismatched hosts visible.

3. CKS Code improvement for the api

https://cloudstack.apache.org/api/apidocs-4.23/apis/createKubernetesCluster.html

extraconfig cannot be used. createKubernetesCluster exposes no extraconfig parameter; CKS
deploys its nodes internally via userVmService.createAdvancedVirtualMachine(...)
(KubernetesClusterResourceModifierActionWorker.java:439) without one; and even if it did, the KVM
agent's addExtraConfigComponent (LibvirtComputingResource.java:3468) only appends raw XML to the
domain via vm.addComp(comp). Since CloudStack already emits a <cpu> element, a second one would be
invalid domain XML rather than an override.

Service offering details do not reach the CPU definition either. HypervisorGuruBase routes service
offering details into the extraConfig map (addServiceOfferingExtraConfiguration, line ~257), not into
the VM details map, so they hit the same additive-XML limitation.

A per-VM detail does override it, but cannot be set at CKS deployment time. The KVM agent honours a
VM detail ahead of the host-wide setting:

// LibvirtComputingResource.java:3310-3314
private CpuModeDef createCpuModeDef(VirtualMachineTO vmTO, int vcpus) {
    final CpuModeDef cmd = new CpuModeDef();
    String cpuMode  = details.get(VmDetailConstants.GUEST_CPU_MODE)  != null ? details.get(VmDetailConstants.GUEST_CPU_MODE)  : guestCpuMode;
    String cpuModel = details.get(VmDetailConstants.GUEST_CPU_MODEL) != null ? details.get(VmDetailConstants.GUEST_CPU_MODEL) : guestCpuModel;

VmDetailConstants.GUEST_CPU_MODE is the string "guest.cpu.mode", and the details map comes from
vm_instance_details (HypervisorGuruBase.java:329). So updateVirtualMachine details[0].guest.cpu.mode=host-model
followed by a stop/start does fix an individual node — but only after the VM exists, i.e. after cluster
creation has already failed. There is no way to supply it up front.

This suggests a small, concrete fix.** CKS already passes VM details through customParameterMap when
creating nodes:

// KubernetesClusterResourceModifierActionWorker.java:408-413
Map<String, String> customParameterMap = new HashMap<String, String>();
customParameterMap.put("rootdisksize", String.valueOf(rootDiskSize));
customParameterMap.put(VmDetailConstants.ROOT_DISK_CONTROLLER, "scsi");

Adding VmDetailConstants.GUEST_CPU_MODE to that map — either unconditionally as a sensible default for
CKS nodes, or as an optional createKubernetesCluster parameter — would let operators deploy working CKS
clusters without touching agent.properties on every KVM host.

[kiranchavala]

cc @Pearl1594 @nvazquez

It is not specific to Calico.**

GOAMD64=v2 adoption is spreading across Go projects, so the same wall will be hit by other CNIs and add-ons over time. The default guest CPU model makes CloudStack guests progressively less able to run current container images.

Note the Kubernetes version is irrelevant to this failure: the Go runtime CPU check runs at process start,
before any contact with the API server. The combination below happens to be Kubernetes 1.37.0, but the
same failure occurs on any Kubernetes version with a Calico build that requires v2.