RHV: After upgrade to RHVH-4.2.7 (RHEL 7.6), libvirt fails to start

Environment

• Red Hat Virtualization 4.2.7
  • Red Hat Enterprise Linux 7.6 Hosts
  • RHVH-4.2.7

Issue

• After upgrading RHVH to 4.2.7, libvirt fails to start.
• After upgrading RHV Host to RHEL 7.6, libvirt fails to start.
• The following error message can be seen:

error : virNetTLSContextLoadCACertListFromFile:550 : Unable to import CA certificate list /etc/pki/vdsm/certs/cacert.pem

Resolution

1. Switch Hosted-Engine to Global Maintenance mode and run engine-setup on it. When asked the question below, ensure it is accepted with Yes:

One or more of the certificates should be renewed, because they expire soon, or include an invalid expiry date, or do not include the subjectAltName extension, which can cause them to be rejected by recent browsers.

For more details see: RHEV-M upgrade asks to renew certificate. What does it mean?.

2. Re-enroll all Hosts via the Administration Portal. Alternatively remove host and add again, no need to re-install the Host OS).

3. Remove the old CA certificate from any API scripts/tools/products or Web Browsers that use it. Then add the new one.

Upgrading to RHV 4.2.8 will ensure the user is warned twice (not once as in 4.2.7 and lower) about not renewing the certificates. This was introduced by RHBA-2019:0121.

Root Cause

The NotBefore date encoding in the CA Certificate is wrong. Up until RHEL 7.5 this was accepted by gnutls/libtasn1 so even if the user did not renew the CA the system was still working fine. But from RHEL 7.6 that incorrect format is not allowed anymore and libvirt fails to start.

RHV's engine-setup command, which is run during upgrades checks for this since several versions ago and asks the user if the certificates should be renewed. If the user never accepts to renew then it is possible that when upgrading to RHEL 7.6 hosts they will stop working as libvirt cannot start with this certificate.

This was investigated in This content is not included.Bugzilla 1648190 - libvirt is unable to start after upgrade to RHEL 7.6 due to cacert.pem.

Diagnostic Steps

Attempting to start libvirt manually, it can be seen that these 2 asserts in gnutls are hit, pointing to ASN1_DER_ERROR error.

# GNUTLS_DEBUG_LEVEL=9 libvirtd --listen
<...>
gnutls[3]: ASSERT: x509.c:311  <-- ASN1_DER_ERROR
gnutls[3]: ASSERT: x509.c:3496

• checking the dates of hosts certificate

  $ openssl x509 -startdate -enddate -noout -in /etc/pki/vdsm/certs/vdsmcert.pem
  

  On the valid cert example we see:

      Validity
          Not Before: Oct 22 04:41:08 2018 GMT
          Not After : Oct 20 04:41:08 2028 GMT
  

  while on the bad one:

      Validity
          Not Before: May 17 16:55:00 2015       <<====
          Not After : May 15 16:55:00 2025 GMT
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.