How do I manually renew Identity Management (IPA) certificates on RHEL 8 after they have expired? (CA-less IPA)
Environment
Red Hat Enterprise Linux 8,Red Hat Enterprise Linux 9Red Hat Identity Management (IPA) v4+
Issue
Identity Management (IdM) installed with or without an embedded Certificate Authority (CA) can use externally signed certificates for the Apache Webserver and 389 Directory Server.
The process is described in the documentation - 11.3. Replacing the web server and LDAP server certificates, but if the certificates are expired on all IPA servers, this approach will not work, because ipa-server-certinstall needs to communicate with other IPA via TLS connection, which will not be verified due to installed certificate invalidity.
For installations with integrated CA, please follow this solution.
Resolution
DISCLAIMERS AND WARNINGS
This procedure was tested and has been verified to work. However, it is a complicated and potentially error-prone procedure, so please do not hesitate to contact Red Hat Technical Support for assistance if you have any questions or concerns.
CAUTION: BE SURE TO CREATE BACKUPS OF THE FOLLOWING DIRECTORIES AND FILES BEFORE BEGINNING.
/etc/dirsrv/slapd-INSTANCE-NAME/
/etc/httpd/alias
/var/lib/certmonger
/var/lib/ipa/certs/
Note: We will assume that 3rd-party CA certificate, installed on the system, is still valid. If it's not, it needs to be installed separately before the steps described.
PROCESS
1. Requesting certificate for LDAP and
HTTP
First, create a certificate signing request (CSR) for the Apache Webserver running on your IdM instances to your 3rd-party CA using the OpenSSL utility. We will be creating a new private key during each of those operations. Please change the server FQDN in subject and subjectAltName according to your environment:
# openssl req -new -newkey rsa:2048 -days 365 -nodes -keyout /var/lib/ipa/private/httpd.key -out /tmp/http.csr -addext "subjectAltName = DNS:server.idm.example.com" -subj '/CN=server.idm.example.com/O=IDM.EXAMPLE.COM'
Same process for the LDAP CSR:
# openssl req -new -newkey rsa:2048 -days 365 -nodes -keyout /etc/dirsrv/slapd-INSTANCE-NAME/ldap.key -out /tmp/ldap.csr -addext "subjectAltName = DNS:server.idm.example.com" -subj '/CN=server.idm.example.com/O=IDM.EXAMPLE.COM'
Submit the CSRs (/tmp/http.csr and /tmp/ldap.csr) to the external CA. The process differs depending on the service to be used as the external CA. After the CA signs the certificate, copy issued certificates to the IPA server.
2. Install the certificate and new key for
HTTPD
Copy the certificate provided to /var/lib/ipa/certs/httpd.crtand the keyfile to /var/lib/ipa/private/httpd.key .
# cp /path/to/httpd.crt /var/lib/ipa/certs/
# cp /path/to/httpd.key /var/lib/ipa/private/
3. Install the certificate for
LDAP
First, list available certificates:
# certutil -L -d /etc/dirsrv/slapd-INSTANCE-NAME/
Default certificate nickname is Server-Cert, but it can be changed. If that is the case, please use present nickname instead of Server-Cert.
Second, we need to remove the old invalid certificate from the NSSDB:
# certutil -D -d /etc/dirsrv/slapd-INSTANCE-NAME/ -n 'Server-Cert' -f /etc/dirsrv/slapd-INSTANCE-NAME/pwdfile.txt
Third, we need to create a PKCS12 file to ease the import process into NSSDB:
# openssl pkcs12 -export -in ldap.crt -inkey ldap.key -out ldap.p12 -name Server-Cert
Next, install the created PKCS12 file to NSSDB with
# pk12util -i ldap.p12 -d /etc/dirsrv/slapd-INSTANCE-NAME/ -k /etc/dirsrv/slapd-INSTANCE-NAME/pwdfile.txt
Finally, check that the new certificate has been successfully imported:
# certutil -L -d /etc/dirsrv/slapd-INSTANCE-NAME/
4. Restart the services
To get the services to use changed certificate, we need to restart them:
# systemctl restart httpd
# systemctl restart dirsrv@INSTANCE
5. Repeat the process for all the replicas.
Since `TLS` talk is impossible until we're finished, we need to perform steps **1-5** on all the replicas that have expired certificates manually.
6. Enroll new certificates to LDAP storage
After the certificates are installed for services, we should also populate LDAP storage of the certificates accordingly.
For `HTTP` certificate:
# ipa-server-certinstall -w --pin=password /var/lib/ipa/private/httpd.key /var/lib/ipa/certs/httpd.crt
Same for LDAP:
# ipa-server-certinstall -d --pin=password /etc/dirsrv/slapd-INSTANCE-NAME/ldap.key /path/to/ldap.crt
After the certificates are installed, once again restart the services:
# systemctl restart httpd
# systemctl restart dirsrv@INSTANCE
Perform actions from this part for each affected replica separately.
Root Cause
CA-less installations with 3rd-party CA need manual certificate renewal and can get expired.
Diagnostic Steps
When running ipa-server-certinstall you get this type of errors:
[Thu Feb 24 11:39:57.133182 2022] [ssl:emerg] [pid 3677355:tid 139932496271680] AH02580: Init: Pass phrase incorrect for key ipa.example.com:443:0
[Thu Feb 24 11:39:57.133248 2022] [ssl:emerg] [pid 3677355:tid 139932496271680] SSL Library Error: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag
[Thu Feb 24 11:39:57.133268 2022] [ssl:emerg] [pid 3677355:tid 139932496271680] SSL Library Error: error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error
[Thu Feb 24 11:39:57.133283 2022] [ssl:emerg] [pid 3677355:tid 139932496271680] SSL Library Error: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag
[Thu Feb 24 11:39:57.133299 2022] [ssl:emerg] [pid 3677355:tid 139932496271680] SSL Library Error: error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error (Type=RSAPrivateKey)
[Thu Feb 24 11:39:57.133315 2022] [ssl:emerg] [pid 3677355:tid 139932496271680] SSL Library Error: error:04093004:rsa routines:old_rsa_priv_decode:RSA lib
[Thu Feb 24 11:39:57.133328 2022] [ssl:emerg] [pid 3677355:tid 139932496271680] SSL Library Error: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag
[Thu Feb 24 11:39:57.133343 2022] [ssl:emerg] [pid 3677355:tid 139932496271680] SSL Library Error: error:0D07803A:asn1 encoding routines:asn1_item_embed_d2i:nested asn1 error (Type=PKCS8_PRIV_KEY_INFO)
[Thu Feb 24 11:39:57.133358 2022] [ssl:emerg] [pid 3677355:tid 139932496271680] AH02311: Fatal error initialising mod_ssl, exiting. See /etc/httpd/logs/error_log for more information
[Thu Feb 24 11:39:57.133368 2022] [ssl:emerg] [pid 3677355:tid 139932496271680] AH02564: Failed to configure encrypted (?) private key ipa.example.com:0, check /var/lib/ipa/private/httpd.key
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.