Virt-who Not Reporting Hypervisor and Guests Information

Environment

• Red Hat Satellite 6.5.3

Issue

• Using the same credentials for 2 different vCenters, but only seeing hypervisors and guests in the Satellite Web UI for one of those virt-who configurations

Resolution

• Removing the duplicate hypervisor from the Satellite inventory will resolve this issue.
  **PLEASE NOTE**
• When virt-who pulls in hypervisors from vCenter, Satellite creates dummy hosts in the inventory that have the string virt-who preceding the hostname. When this is the case, deleting this host from the Web UI is fine, as running the virt-who service again will repopulate this host (if the host still exists in the vCenter inventory).
• If the hypervisor is a RHEL server with OpenStack, RHEV, or Libvirt, please be careful when deleting the host. If the host was created by Satellite, you could accidentally delete actual disks when deleting the hypervisor. This kcs has more information regarding the "Destroy associated VM on host delete" option which you can enable or disable:
  How to delete hosts from Red Hat Satellite 6 WebUI, without actually deleting it from a compute resource?

For more KB articles/solutions related to Virt-who and Virtual Datacenter (VDC) Subscriptions Issues, please refer to the Consolidated Troubleshooting Article for Virt-who and Virtual Datacenter (VDC) Subscriptions Issues

Root Cause

• Virt-who was reporting a hypervisor that was already in the Satellite inventory. The error message was seen in the /var/log/candlepin/candlepin.log:

   2020-07-22 04:21:02,102 [thread=QuartzScheduler_Worker-11] [job=hypervisor_update_ed29a530-a636-4de9-9221-e069de8c2052, org=UTE, csid=] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: duplicate key value violates unique constraint "cp_consumer_hypervisor_ukey"
     Detail: Key (hypervisor_id, owner_id)=(hostname1.example.com, 2c9a1ea066c55fd50166c5603c440001) already exists.
   2020-07-22 04:21:02,102 [thread=QuartzScheduler_Worker-11] [job=hypervisor_update_ed29a530-a636-4de9-9221-e069de8c2052, org=UTE, csid=] ERROR org.candlepin.pinsetter.tasks.HypervisorUpdateJob - HypervisorUpdateJob encountered a problem.
   javax.persistence.PersistenceException: org.hibernate.exception.ConstraintViolationException: could not execute statement

Diagnostic Steps

• Check foreman tasks export to see the Hypervisor tasks that are run during a virt-who refresh. This command will export the tasks:

  # foreman-rake foreman_tasks:export_tasks TASK_SEARCH='' TASK_DAYS=7  

• Stopping virt-who and running it with flags -do will run it in a one shot debug mode. We can capture the output to a file and examine it for errors:

  # systemctl stop virt-who
  # virt-who -do &> /tmp/virt-who-od.txt
  # systemctl start virt-who

• When examining the virt-who debug output, we can see the successful hypervisor to guest mappings for both virt-who configurations:

   2020-07-20 15:53:16,414 [virtwho.destination_-4407671019926252378 INFO] MainProcess(19488):Thread-5 @virt.py:_send_data:651 - Hosts-to-guests mapping for config "virt-who-config-8": 6 hypervisors and 141 guests found

   2020-07-20 15:53:16,392 [virtwho.destination_-6480205103890596683 INFO] MainProcess(19488):Thread-4 @virt.py:_send_data:651 - Hosts-to-guests mapping for config "virt-who-config-9": 56 hypervisors and 758 guests found

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.