How do I manually renew Identity Management (IPA) certificates on RHEL7 after they have expired? (Replica IPA Server)

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Identity Management (IPA) v4

Issue

In normal operation it’s expected that renewal of IPA subsystem certificates is working smoothly. Unfortunately in reality there are sometimes issues to renew those certificates and a manual recovery is necessary in case certificates are already expired.

Resolution

IMPORTANT - Red Hat Enterprise Linux 7.7 and later supports a simplified process of renewing system certificates when IPA is offline. Instructions for renewing expired certificates with the new ipa-cert-fix utility are available here. However, it should be only run on IPA renewal master, because it switches this role to the server it's being run on. If your renewal master is in good shape, make sure your RA agent certificate is correct and use getcert resubmit to renew expired certificates. Revert to ipa-cert-fix on IPA replica only as last resort.

DISCLAIMERS AND WARNINGS
This procedure was tested and has been verified to work. However, it is a complicated and potentially error-prone procedure, so please do not hesitate to contact Red Hat Technical Support for assistance if you have any questions or concerns.

This procedure needs to be run on IPA Replicas with an embedded CA. Separate instructions for renewing IPA certificates on an IPA Master server can be found here.

CAUTION
BE SURE TO CREATE BACKUPS OF THE FOLLOWING DIRECTORIES AND FILES BEFORE BEGINNING.

  • /etc/dirsrv/slapd-REALM/*.db
  • /etc/httpd/alias
  • /var/lib/ipa/ra-agent.{key,pem} (>= RHEL-7.4)
  • /var/lib/certmonger

PROCESS

Step 1: Determine when the IPA certificates were last valid

  • In order for this to work, you will need to set the system clock back to a date and time when the certificates were all still valid. First, we need to stop the NTP service:
# systemctl stop ntpd
  • To find out when the certificates were still valid, run:
# certutil -d  /etc/dirsrv/slapd-<DOMAIN>/ -L -n 'Server-Cert' | grep -i  'after'
# certutil -d  /etc/httpd/alias/ -L -n 'Server-Cert' | grep -i  'after'


RHEL <= 7.3
# certutil -d /etc/httpd/alias/ -L -n 'ipaCert' | grep -i 'after' 

RHEL >= 7.4
# openssl x509 -in /var/lib/ipa/ra-agent.pem -noout -text | grep -i after
  • This tells us how far to set the system clock back. We recommend setting time back at least 24 hours before expiration of the oldest expired certificate.

IMPORTANT: Do not set the date yet, that step comes a little later

Step 2: Verify the validity of the 'ipaCert' (RA) certificate

  • It is very important that the 'ipaCert' (RA) certificate is valid as it is used by the certmonger service to communicate with the CA. If it is expired, certmonger will not be able to renew the certificates it manages. If the RA certificate is expired, you can export a valid RA certificate from the master IPA server and replace the expired copy on the replica.

[on Master]

-- Export the valid RA certificate to ASCII form:

RHEL <= 7.3
# certutil -d /etc/httpd/alias -L -n 'ipaCert' -a > /tmp/newRAcert.crt
  • On systems running RHEL >= 7.4 there is no need to export the RA certificate because it’s already stored as PEM file in /var/lib/ipa/ra-agent.pem. You can just copy this file to /tmp:
# cp  /var/lib/ipa/ra-agent.pem /tmp/newRAcert.crt
  • Copy the RA certificate to the replica IPA server:
# scp /tmp/newRAcert.crt <IPA replica host>:/tmp

[On Replicas]

  • Delete the expired RA certificate:
RHEL <= 7.3
# certutil -d /etc/httpd/alias -D -n 'ipaCert'


RHEL >= 7.4
# rm /var/lib/ipa/ra-agent.pem
  • Import the new RA certificate that was exported from the master IPA server:
RHEL <= 7.3
# certutil -d /etc/httpd/alias -A -n 'ipaCert' -a -i /tmp/newRAcert.crt -t u,u,u


RHEL >= 7.4
# cp /tmp/newRAcert.crt /var/lib/ipa/ra-agent.pem

Step 3: Revert the system clock

Now it's time to set the system clock back to when the certificates were valid with the 'date' command. Again, we recommend setting the clock back at least 24 hours before expiration of the oldest expired certificate. So, for example, if your certificate expired on 10-30-2017 at 10:26, use the following command:

# date 102910262017

Step 4: Renew Directory Server and Apache server certificates

  • Next, we need to renew the Directory Server and the Apache server certificates.
# ipa-getcert list
  • For each of the two Server-Certs Request IDs, run the following command:
# ipa-getcert resubmit -i [Request ID]

Step 5: Return IPA service to Production and restart the world

# ipactl restart

Step 6: Restart certmonger

# systemctl restart certmonger

Step 7: Return the system clock to the current time and confirm it is now current

# systemctl start ntpd
# date

Step 8: Verify that the certificates have been renewed

# getcert list
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.