The Red Hat Virtualization (RHV) 4.3 to 4.4 upgrade must start with z-stream version 4.3.10 or newer

Environment

• Red Hat Virtualization (RHV) 4.3, 4.4

Issue

• During an upgrade to RHVM 4.4, after running engine-backupon the old RHVM 4.3 system and restoring to the new RHVM 4.4 system, engine-setup on the the new RHVM 4.4 system will fail if the old RHVM 4.3 version is earlier than 4.3.10.

Resolution

• Whenever upgrading RHVM to a new version, and especially from 4.3 to 4.4, first bring RHVM up to the latest z-stream release before upgrading to the next release.
• See Chapter 2, Upgrading from 4.3 to Red Hat Virtualization 4.4, in the RHV Upgrade Guide for details.

Root Cause

• RHVM 4.4 depends on Red Hat Enterprise Linux (RHEL) 8.2 or newer.
• RHVM 4.3 depends on RHEL 7 and will not work on RHEL 8.
• Therefore, the RHVM upgrade from 4.3 to 4.4 uses an engine-backup from the old RHVM 4.3 system and a restore to a freshly built RHVM 4.4 system on top of RHEL 8.2 or newer before running engine-setup on the new RHVM 4.4 system to finish the process.
• RHV recently introduced the Cinderlib database as a tech preview, but engine-backup did not back it up until RHV 4.3.10.
• When engine-setup on the new RHVM 4.4 system tries to set up the Cinderlib database, it will fail because it has no Cinderlib configuration from the old RHVM 4.3 system to use.
• This content is not included.Bugzilla 1707225 about backing up the Cinderlib database addressed the problem starting with RHVM 4.3.10.

Diagnostic Steps

• On the original RHVM 4.3 system, use yum list installed | grep rhv or rpm -qa | grep rhv to find the exact RHVM version currently installed.

• If the RHVM version is older than 4.3.10, then after backing up the RHVM 4.3 system and restoring to RMVM 4.4, the engine-setup program will fail with an error similar to below.

[root@rhevm2017 ~]# engine-setup
[ INFO  ] Stage: Initializing
[ INFO  ] Stage: Environment setup
          Configuration files: /etc/ovirt-engine-setup.conf.d/10-packaging-wsp.conf, /etc/ovirt-engine-setup.conf.d/10-packaging.conf, /etc/ovirt-engine-setup.conf.d/20-setup-ovirt-post.conf
          Log file: /var/log/ovirt-engine/setup/ovirt-engine-setup-20200913154825-m53faz.log
          Version: otopi-1.9.2 (otopi-1.9.2-1.el8ev)
[ ERROR ] Failed to execute stage 'Environment setup': Cannot connect to ovirt cinderlib database using existing credentials: ovirt_cinderlib@localhost:5432
[ INFO  ] Stage: Clean up
          Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20200913154825-m53faz.log
[ INFO  ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20200913154834-setup.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Execution of setup failed
[root@rhevm2017 ~]#

• The best recovery in this case is, wipe the target RHEL 8.2 system with RMVM 4.4 and rebuild it from bare metal.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.