Instances are not manageable after upgrading deployment from RHOSP16.2.0 to RHOSP16.2.z

Solution Verified - Updated

Environment

  • Red Hat OpenStack Platform 16.2

Issue

  • After upgrading the deployment from RHOSP16.2.0 to RHOSP16.2.z, each instance shows OS-EXT-STS:power_state: NOSTATE
  • Operations like stop or hard reboot for these instances consistently fail

Resolution

This KCS is published for public reference. Please contact Red Hat Technical Support before applying the solution

  1. Deploy RHOSP16.2.0 images for nova and libvirt services on compute nodes.

  • Modify the following three files to use RHOSP16.2.0 images

    • /var/lib/tripleo-config/container-startup-config/step_3/hashed-nova_libvirt.json
    • /var/lib/tripleo-config/container-startup-config/step_3/hashed-nova_virtlogd.json
    • /var/lib/tripleo-config/container-startup-config/step_5/hashed-nova_compute.json

  • Recreate nova/libvirt containers with RHOSP16.2.0 images

    • $sudo paunch --debug apply --file /var/lib/tripleo-config/container-startup-config/step_3 --config-id tripleo_step3 --managed-by tripleo-Compute
    • $ sudo paunch --debug apply --file /var/lib/tripleo-config/container-startup-config/step_5 --config-id tripleo_step5 --managed-by tripleo-Compute

NOTE: Replace Compute according to the role actually used to deploy the compute node.

  1. After this stage, the power state of each instance should be restored. If there is any instances which be stopped then stop these by openstack server stop <server id>

  2. Check whether the nova-conductor container in all controller nodes include the fix for This content is not included.bz 2109350

NOTE: The package is not yet released at the time of writing. Please reach out to Red Hat Technical Support to get a hotfix package

  1. Enable the workaround option( [workaround] skip_hypervisor_version_check_on_lm = True ) in nova.conf in all controller nodes to allow migration from RHOSP16.2.0 libvirt to RHOSP16.2.z libvirt. The following environment file can be used to configure the option by RHOSP Director.
parameter_defaults:
  ControllerExtraConfig:
    nova::config::nova_config:
      workarounds/skip_hypervisor_version_check_on_lm:
        value: true

NOTE: Executing the deployment replaces RHOSP16.2.0 container images by RHOSP16.2.z container images.

  1. Prepare a compute node with RHOSP16.2.z container images

  2. Live migrate instance from the node with RHOSP16.2.0 containers to the node with RHOSP16.2.z.

NOTE: Use nova --os-compute-api-version 2.66 live-migration <server> <host> --force in case scheduler filer should be bypassed.

  1. Once all active instances are migrated, revert the change in json files made earlier and rerun pauch to deploy RHOSP16.2.z container images again.

  2. Start instances on that compute node wit RHOSP16.2.z container images

Root Cause

  • The latest RHOSP16.2.0 containers include libvirt packages from RHEL8.5 (libvirt 7.6) advanced virtualization instead of libvirt packages from RHEL8.4 (libvirt 7.0).
  • This was fixed later in RHOSP16.2.1 release
  • Instances launched by libvirt 7.6 is not compatible with libvirt 7.0. So the instances created/started in RHOSP16.2.0 is no longer manageable after update to RHOSP16.2.z.

Diagnostic Steps

  • The following error repeatedly appears in /var/log/containers/libvirt/libvirtd.log
2022-07-21 12:34:56.789+0000: 1234567: error : virDomainAudioDefParseXML:13818 : unsupported configuration: unknown audio type 'none'
  • The deployment was using the affected version of container images, described in the Root Cause section.
SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.