How to Clean Up a Failed Self-hosted Engine Deployment?

Environment

• Red Hat Enterprise Virtualization (RHEV) >= 3.6
• Self-hosted-engine on FibeChannel or iSCSI

Issue

• If a self-hosted engine installation was interrupted and the deployment is re-run, the following error is shown:
  Failed to connect to broker, the number of errors has exceeded the limit (1).

• In the installation logs (/var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-*.log) you will see something like:

    2015-09-01 07:23:13 DEBUG otopi.context context._executeMethod:152 method exception
    Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod
      File "/usr/share/ovirt-hosted-engine-setup/plugins/ovirt-hosted-engine-setup/vm/configurevm.py", line 112, in _late_setup
    RuntimeError: Cannot setup Hosted Engine with other VMs running
    2015-09-01 07:23:13 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Environment setup': Cannot setup Hosted Engine with other VMs running
  

Resolution

In order to re-run the self-hosted engine deployment successful, the hosted-engine needs to be stopped and the related configuration files and keys need to be cleaned up. In addition to this, a clean up of the self-hosted engine storage domain is required if it contains existing data.

1. Power off the hosted-engine virtual machine (engine) if it is running:

    host# hosted-engine --vm-poweroff
   

   • check for leftover vm-definition

       host# vdsClient -s 0 getAllVmStats
       506d17a0-2fc0-4db1-878e-82e9917bff4a
               Status = Down
       [...]
       host# vdsClient -s 0 destroy 506d17a0-2fc0-4db1-878e-82e9917bff4a
       Machine destroyed
     

2. Stop the high availability services and the VDSM service:

    host# systemctl stop ovirt-ha-agent ovirt-ha-broker vdsmd
   

3. Check the hosted-engine storage domain and remove any data. THIS WILL DELETE ALL DATA OF THE HOSTED-ENGINE !!

   • find the device where hosted-engine is located on

       host# vgs -o devices $(awk -F= '/sdUUID/ {print $2}' /etc/ovirt-hosted-engine/hosted-engine.conf)
         Devices
         /dev/mapper/hosted-engine(0)
       [...]
         /dev/mapper/hosted-engine(442)
     

   • remove the volume-group and wipe the physical-volume header

       host# vgremove $(awk -F= '/sdUUID/ {print $2}' /etc/ovirt-hosted-engine/hosted-engine.conf)
     
       host# pvremove /dev/mapper/hosted-engine
     

4. Remove the hosted-engine related configuration files and keys:

    host# /bin/rm /etc/ovirt-hosted-engine/hosted-engine.conf
    host# /bin/rm /etc/ovirt-hosted-engine/answers.conf
    host# /bin/rm /etc/vdsm/vdsm.conf
    host# /bin/rm /etc/pki/vdsm/*/*.pem
    host# /bin/rm /etc/pki/CA/cacert.pem
    host# /bin/rm /etc/pki/libvirt/*.pem
    host# /bin/rm /etc/pki/libvirt/private/*.pem
   

5. Reboot the hosted-engine host.
   Note: The reboot is needed to make sure all the connections to the storage are cleaned before the next attempt.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.