[RHV] ovirt-provider-ovn certs aren't renewed with engine-setup

Environment

• Red Hat Virtualization 4.x

Issue

After renewal of expired certificates for RHV, there is an issue with our OVN:

2022-04-01 11:38:33,491 root Error during SSO authentication invalid_grant : The provided authorization grant for the auth code has expired.
Traceback (most recent call last):

Or:

Apr 1 22:06:27 manager ovsdb-server[1478]: ovs|18062|reconnect|WARN|ssl:[::ffff:10.1.1.13]:43034: connection dropped (Protocol error)
Apr  1 22:06:27 manager ovsdb-server[1478]: ovs|18063|reconnect|WARN|ssl:[::ffff:10.1.1.15]:45404: connection dropped (Protocol error)

Resolution

As engine-setup in previous versions did not renew OVS certificates, they need to be manually renewed as follows:

1. Find the subject name of the existing certificates, e.g.:

[root@rhvm2 ~]# openssl x509 -in /etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer -noout -subject
subject=C = US, O = rhvmgmt.tamlab.rdu2.redhat.com, CN = rhvm2.rhvmgmt.tamlab.rdu2.redhat.com

2. Generate new certs

Note: The specified --password=mypass parameter for all commands it is literally the password that needs to be used. Do not change it.

Note also that the subject needs to be in the format /type0=value0/type1=value1/type2=...

[root@rhvm2 ~]# /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh --name="ovirt-provider-ovn" --password=mypass --subject="/C=US/O=rhvmgmt.tamlab.rdu2.redhat.com/CN=rhvm2.rhvmgmt.tamlab.rdu2.redhat.com" --keep-key

[root@rhvm2 ~]# /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh --name="ovn-ndb" --password=mypass --subject="/C=US/O=rhvmgmt.tamlab.rdu2.redhat.com/CN=rhvm2.rhvmgmt.tamlab.rdu2.redhat.com" --keep-key

[root@rhvm2 ~]# /usr/share/ovirt-engine/bin/pki-enroll-pkcs12.sh --name="ovn-sdb" --password=mypass --subject="/C=US/O=rhvmgmt.tamlab.rdu2.redhat.com/CN=rhvm2.rhvmgmt.tamlab.rdu2.redhat.com" --keep-key

3. Restart services

# systemctl restart ovirt-provider-ovn.service
# systemctl restart ovn-northd.service

Root Cause

• RHV-M Certs had expired and were renewed with engine-setup. This does not renew ovirt-provider-ovn, ovn-ndb and ovn-sdb certificates.
• Note that this issue is fixed in 4.5.2, Errata RHSA-2022:6393

Diagnostic Steps

2022-04-01 11:38:33,491 root Error during SSO authentication invalid_grant : The provided authorization grant for the auth code has expired.
Traceback (most recent call last):
  File "/usr/share/ovirt-provider-ovn/handlers/base_handler.py", line 138, in _handle_request
    method, path_parts, content
  File "/usr/share/ovirt-provider-ovn/handlers/selecting_handler.py", line 175, in handle_request
    return self.call_response_handler(handler, content, parameters)
  File "/usr/share/ovirt-provider-ovn/handlers/neutron.py", line 33, in call_response_handler
    TOKEN_HTTP_HEADER_FIELD_NAME, '')):
  File "/usr/share/ovirt-provider-ovn/auth/plugin_facade.py", line 31, in validate_token
    return auth.core.plugin.validate_token(token)
  File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 36, in validate_token
    return self._is_user_name(token, _admin_user_name())
  File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/authorization_by_username.py", line 47, in _is_user_name
    timeout=AuthorizationByUserName._timeout())
  File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 131, in get_token_info
    timeout=timeout
  File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 55, in wrapper
    _check_for_error(response)
  File "/usr/share/ovirt-provider-ovn/auth/plugins/ovirt/sso.py", line 181, in _check_for_error
    result['error'], details))
auth.errors.Unauthorized: Error during SSO authentication invalid_grant : The provided authorization grant for the auth code has expired.

[root@ovirt-engine1 ~]# openssl x509 -in /etc/pki/ovirt-engine/certs/ovirt-provider-ovn.cer -noout -dates
notBefore=Feb 24 10:48:50 2021 GMT
notAfter=Mar 30 10:48:50 2022 GMT

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.