How to manually renew RHV host SSL certificate if expired?

Environment

• Red Hat Virtualization (RHV) 4.x

Issue

• RHV host SSL certificate expired and the host changed to Not Responding state. It is not possible to manage the VMs running on this host using the RHV-M portal.
• Is it possible to manually renew the host certificates so that users can move the workload to a different host?

Resolution

IMPORTANT: The recommended method to renew the host certificate is either:

• Use the Enroll Certificate feature in the RHV-M portal as documented in KCS How do I keep Red Hat Virtualization (RHV) host and manager certificates current.
• Or remove and re-add the host.
• From RHV 4.4 SP1, it is possible to use the Enroll Certificate from the RHV-M administrative portal when the host is even in NonResponsive status after the certificate is expired.
• If hosts are in NonResponsive status in an RHV-M 4.4 environment, the recommended approach is to upgrade the RHV-M to 4.4 SP1 and use the Enroll Certificate feature instead of manually renewing certificates. The upgrade instructions are available at Updating the Red Hat Virtualization Manager.
• If all hosts are expired and the Manager (RHVM) is down, use KCS How to start hosted-engine VM if the host certificates are expired instead. This provides steps to retrieve the required CA certificate files

Only follow the below steps for a host already in not responding status and if it's not possible to upgrade the RHV-M to 4.4 SP1 or if there is no way to get a maintenance window to shut down VMs running on that host. After following the manual steps below and getting the host status to UP, Red Hat still recommends customers follow the automated steps above to ensure the host has the proper certificates.

Note: In some cases not only host certificates expire, but also the CA and the manager's. In such a case follow KCS How do I keep Red Hat Virtualization (RHV) host and manager certificates current to automatically renew manager certificates with engine_setup.

For an automated solution to renew host certificates, please see KCS Automated Renewal of expired RHV Certs

Renew a host certificate manually through the following steps.

1. Copy the key to the manager.

[HOST]# scp /etc/pki/vdsm/keys/vdsmkey.pem root@<RHV-M FQDN OR IP>:/tmp/vdsmkey.pem

2. Create a CSR for the host using the key.

[MANAGER]# openssl req -new -key /tmp/vdsmkey.pem -out /tmp/test_host_vdsm.csr -passin "pass:mypass" -passout "pass:mypass" -batch -subj "/"

3. On the hypervisor, find the subject of the old certificate.

[HOST]# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject

4. On the manager, sign the CSR using the engine CA.

RHV 4.3 and above:

• Determine whether the host was added to the RHV manager using an IP address or a FQDN:

[MANAGER]# /usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "select host_name from vds_static where vds_name='name-of-the-host';"
             host_name
-----------------------------------
       rhvh1.example.com
(1 row)

• Sign the CSR. Make sure that the subj is replaced from step [3] as well as OVIRT_SAN with the output obtained from the previous step. For OVIRT_SAN, substitute it with DNS:FQDN if the host was added using an FQDN or with IP:ipaddress if the host was added using an IP address.

[MANAGER]# cd /etc/pki/ovirt-engine/
[MANAGER]# OVIRT_KU="" OVIRT_EKU="" OVIRT_SAN="DNS:rhvh1.example.com"  openssl ca -batch -policy policy_match -config openssl.conf -cert ca.pem -keyfile  private/ca.pem -days +398 -in  /tmp/test_host_vdsm.csr -out /tmp/test_host_vdsm.cer -startdate "$(date --utc --date "now -1 days" +"%y%m%d%H%M%SZ")" -subj "/C=US/O=Test/CN=test.redhat.com" -utf8 -extfile cert.conf -extensions v3_ca_san

Versions below RHV 4.3:

• Sign the CSR. Make sure that the subj is replaced from step [3]:

[MANAGER]# cd /etc/pki/ovirt-engine/
[MANAGER]# openssl ca -batch -policy policy_match -config openssl.conf -cert ca.pem -keyfile  private/ca.pem -days +398 -in  /tmp/test_host_vdsm.csr -out /tmp/test_host_vdsm.cer -startdate "$(date --utc --date "now -1 days" +"%y%m%d%H%M%SZ")" -subj "/C=US/O=Test/CN=test.redhat.com" -utf8

5. Copy back the signed certificate to /etc/pki/vdsm/certs/vdsmcert.pem.

[HOST]# scp root@<RHV-M FQDN OR IP>:/tmp/test_host_vdsm.cer /etc/pki/vdsm/certs/vdsmcert.pem

6. Copy the certificate to libvirt.

[HOST]# cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem
[HOST]# cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/libvirt/clientcert.pem

7. NOTE Its likely the libvirt-migrate certs have also expired. Please follow KCS Regenerate RHV host certificate for libvirt migrations prior to moving to the next step.

8. Disable the power management of the host from the Admin Portal and restart both libvirtd and vdsmd services from the host in question.

[HOST]# systemctl restart libvirtd vdsmd

9. Once the host is up, live migrate the VMs and enroll the certificate of the host from the manager portal.

Compute -> Hosts -> Select the host -> Management -> Maintenance
Installation -> Enroll Certificate

Root Cause

• The host SSL certificate expired which broke the communication between RHV manager and hypervisor.

Diagnostic Steps

• Check the expiry date of the certificate using the command:

    # openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.