Regenerate RHV host certificate for libvirt migrations

Solution Verified - Updated

Environment

  • Red Hat Virtualization (RHV) 4.4

Issue

  • The libvirt certificate for live migrations has expired, so we cannot live migrate the VMs to re-enroll the host certificates.
  • We need to re-enroll the certificate manually.

Resolution

  • In the RHV-M machine verify that you have the host's signing request at:
    /etc/pki/ovirt-engine/requests-qemu/<Host FQDN OR IP>.req.

  • Find the subject of the old certificate from the host.

    [HOST]# openssl x509 -in /etc/pki/vdsm/libvirt-migrate/server-cert.pem  -noout -subject
subject=O = example.com, OU = qemu, CN = rhvh1.example.com

  • Sign the signing request from the manager:

    NOTES:
    - Replace rhvh1.example.com with host FQDN and subject with the output obtained from the above step.
    - If CN uses the IP address of the host, use the IP option in the SAN: --san="IP:1.2.3.4"

    [MANAGER]# /usr/share/ovirt-engine/bin/pki-enroll-request.sh \
        --name=rhvh1.example.com \
        --subject="/O=example.com/OU=qemu/CN=rhvh1.example.com" \
        --san="DNS:rhvh1.example.com" \
        --days=3650 \
        --ca-file=qemu-ca  \
        --cert-dir=certs-qemu \
        --req-dir=requests-qemu

    The pki-enroll-request.sh script is different with RHV 4.3.
    - name="$1"
    - subject="$2"
    - days="$3"
    - ovirt_ku="$4"
    - ovirt_eku="$5"
    - ovirt_san="$6"
    Here is an example for using it with RHV 4.3.

    [4.3 MANAGER] # /usr/share/ovirt-engine/bin/pki-enroll-request.sh \
        --name=rhv1.example.com \
        --subject="/CN=rhv1.example.com" \
        --days=3650 \
        --ku="" \
        --eku="" \
        --san="DNS:rhv1.example.com"

  • Copy the generated cert to the host:

    [MANAGER]# scp -i /etc/pki/ovirt-engine/keys/engine_id_rsa /etc/pki/ovirt-engine/certs-qemu/rhvh1.example.com.cer root@rhvh1.example.com:/etc/pki/vdsm/libvirt-migrate/server-cert.pem

  • Temporarily disable power management in the host and restart the libvirtd service:

        [HOST]# systemctl restart libvirtd mom-vdsm ovirt-imageio vdsmd supervdsmd

Root Cause

The hosts have to be in maintenance mode before re-enrolling their certificates. To do so, all VMs have to be migrated away however, if the certificates have already expired, the live migrations are not possible.

Diagnostic Steps

  • The host certificate for libvirt is expired:
# openssl x509 -in /etc/pki/vdsm/libvirt-migrate/server-cert.pem -noout -dates
  • In vdsm.log we can see these messages for the failed migrations:
2022-05-05 13:42:46,219+0200 ERROR (migsrc/4236b181) [virt.vm] (vmId='4236b181-b1ab-4302-8282-a97e2a317105') Failed to migrate (migration:460)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 441, in _regular_run
    time.time(), machineParams
  File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 530, in _startUnderlyingMigration
    self._perform_with_conv_schedule(duri, muri)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 619, in _perform_with_conv_schedule
    self._perform_migration(duri, muri)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/migration.py", line 548, in _perform_migration
    self._migration_flags)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 159, in call
    return getattr(self._vm._dom, name)(*a, **kw)
  File "/usr/lib/python3.6/site-packages/vdsm/virt/virdomain.py", line 101, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1944, in migrateToURI3
    if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
libvirt.libvirtError: internal error: unable to execute QEMU command 'object-add': The server certificate /etc/pki/vdsm/libvirt-migrate/server-cert.pem has expired
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.