Workaround for a libvirt version-compat issue (bug 2109350) when updating RHOSP 16.2.0

Solution Unverified - Updated

Environment

  • Red Hat OpenStack Platform 16.2.0

Issue

  • RHOSP16.2.0 has a known issue which makes all instances unmanageable after minor update. This problem occurs only in update from 16.2.0 to a newer release such as 16.2.4

Resolution

Follow the steps below to check whether the deployment is affected by the issue

Check the libvirt package in the nova_libvirt container in all compute nodes.

$ sudo podman exec nova_libvirt rpm -q libvirt

  • If the libvirt version is 7.0, the deployment IS NOT affected by the bug. No workaround is needed.

  • If the libvirt version is 7.6, the deployment IS affected by the bug.

If the deployment is affected, follow the below procedure during updating compute/hci nodes, to keep instances manageable

NOTE: This workaround requires the new option which is available since RHOSP16.2.4 release. Please DO NOT update the affected deployment until RHOSP16.2.4 is released

  1. Enable the wrokaround option and restart nova-conductor in all controller nodes
$ sudo sudo crudini --set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf  \
    workarounds skip_hypervisor_version_check_on_lm True
$ sudo systemctl restart tripleo_nova_conductor.service

  1. Prepare empty compute nodes without any instances running. Instances on these nodes should be migrated to the other compute nodes or shut off.

  2. Update the compute nodes without running instances to RHOSP16.2.latest . Use the --limit option to update only selective compute nodes

$ openstack overcloud update run --limit compute-0,compute-1

  1. Select the compute nodes being updated next. All instances on these nodes should be migrated to the other compute nodes or shut off.

  2. Update the compute nodes without running instances to RHOSP16.2.latest using the --limit option.

$ openstack overcloud update run --limit compute-2,compute-3

  1. Repeat step 4 and 5 until all compute/hci nodes are updated to to RHOSP16.2.latest

  2. Disable the wrokaround option and restart nova-conductor in all controller nodes

$ sudo sudo crudini --set /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf  \
    workarounds skip_hypervisor_version_check_on_lm False
$ sudo systemctl restart tripleo_nova_conductor.service
SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.