Why VM live migration is failing with error "netplugin failed: /var/lib/cni/bin/cnv-bridge: /lib64/libc.so.6: version `GLIBC_2.32' not found"?

Solution Verified - Updated

Environment

  • OpenShift Virtualization 4.12.

Issue

  • The VM live migration was failing when the worker pool was upgraded during the EUS upgrade from 4.12 to 4.14. Destination virt launcher pod fails to start and logged below message:
4m10s       Warning   FailedCreatePodSandBox       pod/virt-launcher-centos-stream8-fc4tl5xpdvmr06pq-s7hqm   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-centos-stream8-fc4tl5xpdvmr06pq-s7hqm_my-cnv_7ac1c96b-7b9f-4da4-8712-8d1b914d7371_0(88ece225a8dec08a2ec02207ddb55a4ff843af486aad6ad1c42e574ddf2ae073): error adding pod my-cnv_virt-launcher-centos-stream8-fc4tl5xpdvmr06pq-s7hqm to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [my-cnv/virt-launcher-centos-stream8-fc4tl5xpdvmr06pq-s7hqm/7ac1c96b-7b9f-4da4-8712-8d1b914d7371:br1]: error adding container to network "br1": netplugin failed: "/var/lib/cni/bin/cnv-bridge: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/lib/cni/bin/cnv-bridge)\n/var/lib/cni/bin/cnv-bridge: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/lib/cni/bin/cnv-bridge)\n"

Resolution

  • Edit the NetworkAttachmentDefinition and change the type from cnv-bridge to bridge.

Root Cause

  • During the EUS upgrade, the worker mcp will be paused. So the worker nodes will stay in 4.12 until the master and other operators are upgraded to 4.14. The OpenShift Virtualization will be also get upgraded to 4.14 before worker nodes which also upgrades kube-cni-linux-bridge-plugin. This installs new versions of the cnv-bridge and cnv-tuning in the worker nodes:
  • Since the worker nodes are still 8.x version, the bridge binary will fail in these node because it don't have the new GLIBC:
openshift-worker-mytemp-0 ~]# /var/lib/cni/bin/cnv-bridge
/var/lib/cni/bin/cnv-bridge: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by /var/lib/cni/bin/cnv-bridge)
/var/lib/cni/bin/cnv-bridge: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by /var/lib/cni/bin/cnv-bridge)

openshift-worker-mytemp-0 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux CoreOS release 4.12
SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.