How do I manually renew Identity Management (IPA) certificates on RHEL6 after they have expired? (Master IPA Server)

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 6
  • Red Hat Identity Management (IPA) v3

Issue

In normal operation it’s expected that renewal of IPA subsystem certificates is working smoothly starting with Red Hat Enterprise Linux 6.4 . Unfortunately in reality there are sometimes issues to renew those certificates and a manual recovery is necessary in case certificates are already expired.

Resolution

DISCLAIMERS AND WARNINGS
This procedure was tested and has been verified to work. However, it is a complicated and potentially error-prone procedure, so please do not hesitate to contact Red Hat Technical Support for assistance if you have any questions or concerns.

This procedure needs to be run on an IPA Master with an embedded CA. Separate instructions for renewing IPA certificates on IPA Replica servers can be found here. For RHEL7 instructions, please visit this page.

CAUTION
BE SURE TO CREATE BACKUPS OF THE FOLLOWING DIRECTORIES AND FILES BEFORE BEGINNING.

  • /etc/dirsrv/slapd-REALM/*.db
  • /etc/dirsrv/slapd-PKI-IPA/*.db
  • /etc/httpd/alias
  • /etc/pki-ca/CS.cfg
  • /var/lib/certmonger
  • /var/lib/pki-ca/alias

PROCESS

Step1: List certificates to confirm tracking

  • Run the command getcert list. Use the output from the above command to confirm that a total of 8 certificates are being tracked for renewal and if there are, skip to Step 2.

Hint - the first line of output from the getcert list command will indicate the number of certificates being tracked:

Number of certificates and requests being tracked: 8.

  • If less than 8 certificates are being tracked, run the following commands to add tracking for the remainder of the certificates and then proceed to Step 2 (note that IPA v3 servers normally already have tracking in place for all 8 certificates):

  • Obtain INTERNAL PIN to access the NSS database(used in the below getcert start-tracking commands):

# grep 'internal=' /var/lib/pki-ca/conf/password.conf
  • Add tracking:
# getcert start-tracking -d /var/lib/pki-ca/alias -n "auditSigningCert cert-pki-ca" -c dogtag-ipa-renew-agent -P <<internal PIN>> -B /usr/lib64/ipa/certmonger/stop_pkicad  -C '/usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"'

# getcert start-tracking -d /var/lib/pki-ca/alias -n "ocspSigningCert cert-pki-ca" -c dogtag-ipa-renew-agent -P <<internal PIN>> -B /usr/lib64/ipa/certmonger/stop_pkicad -C '/usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"'

# getcert start-tracking -d /var/lib/pki-ca/alias -n "subsystemCert cert-pki-ca" -c dogtag-ipa-renew-agent -P <<internal PIN>>  -B /usr/lib64/ipa/certmonger/stop_pkicad  -C '/usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"'

# getcert start-tracking -d /var/lib/pki-ca/alias -n "Server-Cert cert-pki-ca" -c dogtag-ipa-renew-agent -P <<internal PIN>> -C '/usr/lib64/ipa/certmonger/restart_dirsrv "<<LDAP Domain Name>>"' 

# getcert start-tracking -d /etc/httpd/alias -n ipaCert -c dogtag-ipa-renew-agent -p /etc/httpd/alias/pwdfile.txt -C /usr/lib64/ipa/certmonger/restart_httpd
  • Here is an example:
# grep 'internal=' /var/lib/pki-ca/conf/password.conf
internal=318909843122

# getcert start-tracking -d /var/lib/pki-ca/alias -n "auditSigningCert cert-pki-ca" -c dogtag-ipa-renew-agent -P 318909843122 -B /usr/lib64/ipa/certmonger/stop_pkicad  -C '/usr/lib64/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"'

# getcert start-tracking -d /var/lib/pki-ca/alias -n "ocspSigningCert cert-pki-ca" -c dogtag-ipa-renew-agent -P 318909843122 -B /usr/lib64/ipa/certmonger/stop_pkicad -C '/usr/lib64/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"'

# getcert start-tracking -d /var/lib/pki-ca/alias -n "subsystemCert cert-pki-ca" -c dogtag-ipa-renew-agent -P 318909843122  -B /usr/lib64/ipa/certmonger/stop_pkicad  -C '/usr/lib64/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"'

# getcert start-tracking -d /var/lib/pki-ca/alias -n "Server-Cert cert-pki-ca" -c dogtag-ipa-renew-agent -P 318909843122 -C '/usr/lib64/ipa/certmonger/restart_dirsrv HG-EXAMPLE-COM'

# getcert start-tracking -d /etc/httpd/alias -n ipaCert -c dogtag-ipa-renew-agent -p /etc/httpd/alias/pwdfile.txt -C /usr/lib64/ipa/certmonger/restart_httpd
  • Run the getcert list command again to confirm that 8 certificates are now being tracked.

Step 2: Determine when the certificates were last valid

  • In order for this to work, you will need to set the system clock back to a date and time when the certificates were all still valid. First, we need to stop the NTP service:
# /sbin/service ntpd stop
  • To find out when the certificates were still valid, run the following command:
# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"
           do
             echo $nickname
             certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" | grep -i after
           done
  • This tells us how far to set the system clock back in time.

Step 3: Revert the system clock

  • Now it's time to set the system clock back to when the certificates were valid. We recommend setting the date back at least 24 hours prior to the expiration date of the earliest expiring certificate from Step 2. For example, if your certificate expired on 10-30-2017 at 10:26, use the following command:
# date 102910262017

Step 4: Renew the certificates

  • With the system clock now set back in time to allow the expired certificates to be valid, we need to force renewal on all of the certificates. Restart the certmonger service to resubmit renewal requests for the expired certificates:
# service certmonger restart
  • This will renew the CA subsystem certificates, but the SSL server certificates for the Apache and 389 Directory Server services will have failed to renew. We need to do a little more work to get these certificates renewed so that the IPA CA will be fully operational. Running the getcert list command again should confirm that these certificates still need to be updated.

Step 5: Verify and fix CA subsystem certificate trust permissions

  • The trust attributes of the internal CA subsystem certificates should be set as in the below example (note that the ordering of the certificates may differ - what is important is that the trust attributes are correct). Pass the list ('-L') option to the certutil command to display the attributes that are currently set:
# certutil -d /var/lib/pki-ca/alias -L
Certificate Nickname                                 Trust Attributes         SSL,S/MIME,JAR/XPI

ocspSigningCert cert-pki-ca                  u,u,u
subsystemCert cert-pki-ca                     u,u,u
caSigningCert cert-pki-ca                       CTu,Cu,Cu
Server-Cert cert-pki-ca                            u,u,u
auditSigningCert cert-pki-ca                u,u,Pu

Note that the CA subsystem will NOT run correctly if the trust attributes of its internal certificates are incorrect.

  • If the trust attributes for each certificate are correct, proceed to the next step. If not, set the correct trust attributes with the following command:
# certutil -M -n "<Certificate Nickname>" -d /var/lib/pki-ca/alias -t <Trust attributes>
  • For example, to correct the trust attributes for the auditSigningCert cert-pki-ca certificate, use the following command:
# certutil -M -n "auditSigningCert cert-pki-ca" -d /var/lib/pki-ca/alias -t u,u,Pu
  • Repeat the above certutil command to correct any other certificates that have incorrect attributes set.

Step 6: Update subsystem certificate blobs in the IPA CA configuration file

  • The IPA CA configuration file contains Base64-encoded copies of the CA subsystem certificates. You'll need to confirm that the configuration file contains the current certificates and update as needed.

  • First, extract the individual CA subsystem certificates to single line text files in the /tmp directory:

# for nickname in "auditSigningCert cert-pki-ca" "ocspSigningCert cert-pki-ca" "subsystemCert cert-pki-ca" "Server-Cert cert-pki-ca"; do certutil -L -d /var/lib/pki-ca/alias -n "${nickname}" -a | sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE-----$/b2;H;b1};:2;${x;s/\s//g;p}'  > /tmp/"${nickname}.pem"; done
  • Next, review the file /etc/pki-ca/CS.cfg and find the blobs for the CA subsystem certificates by their nicknames. Here are the nicknames and values:
nicknamevalue
auditSigningCert cert-pki-caca.audit_signing.cert
ocspSigningCert cert-pki-caca.ocsp_signing.cert
subsystemCert cert-pki-caca.subsystem.cert
Server-Cert cert-pki-caca.sslserver.cert

  • Compare the certificate blobs in CS.cfg with the respective copies that you extracted to /tmp earlier; they should match and if so, you can delete the single line text files that you created in /tmp and proceed to the next step.

  • If they do NOT match. Stop the CA service:

# service pki-cad stop

  • Edit /etc/pki-ca/CS.cfg, find the certificate entry that needs to be updated and replace the blobs with the copies that you extracted to single-line files in /tmp. Be sure to back up /etc/pki-ca/CS.cfg' before editing it! A corrupted copy of this file will cause the CA to fail to start.

  • Restart the CA and verify that it is running:

# service pki-cad start
# service pki-cad status

Step 7: Test CA operation

  • To test the operation of the CA, use the following command:
# curl --cacert /etc/ipa/ca.crt -v https://`hostname`:9444/ca/ee/ca/getCertChain
  • You should receive a response from the CA similar to the following example:
           About to connect() to ipaserver-1.example.com port 9444 (#0)
                Trying 192.168.122.137... connected
           Connected to ipaserver-1.example.com (192.168.122.137) port 9444 (#0)
            Initializing NSS with certpath: sql:/etc/pki/nssdb
                CAfile: /etc/ipa/ca.crt
           CApath: none
           SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
           Server certificate:
 	              subject: CN=ipaserver-1.example.com,O=EXAMPLE.COM
 	              start date: Apr 14 19:11:09 2016 GMT
 	              expire date: Apr 04 19:11:09 2018 GMT
 	              common name: ipaserver-1.example.com
 	               issuer: CN=Certificate Authority,O=EXAMPLE.COM
            > GET /ca/ee/ca/getCertChain HTTP/1.1
            > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
            > Host: ipaserver-1.example.com:9444
            > Accept: */*
            > 
            < HTTP/1.1 200 OK
            < Server: Apache-Coyote/1.1
            < Content-Type: application/xml
            < Content-Length: 1462
            < Date: Wed, 03 Aug 2016 16:36:00 GMT
            < 
            [...]

Step 8: Verify and update IPA agent certificate

  • The IPA agent certificate (ipaCert) is used to authenticate with the CA for certificate operations and is stored in LDAP. This entry must be stored correctly in order for communication with the CA to occur.

  • Start by determining the new serial number for the agent certificate:

# certutil -L -d /etc/httpd/alias -n ipaCert | grep -i serial
  • Next, verify what data is stored in LDAP for this entry:
# ldapsearch -x -h localhost -p 7389 -D 'cn=directory manager' -W -b uid=ipara,ou=People,o=ipaca
  • Compare the serial number for the agent certificate in the description attribute with the current one. The serial number is the 2nd number. The format of the description line is:
2;[serial number];[issuer subject];[subject]

  • If the serial numbers are identical, the entry has already been updated. Proceed to the next step.

  • If the serial numbers do NOT match, run the following commands. Extract the Base64-encoded value of the agent certificate to a single line file in /tmp:

# certutil -L -d /etc/httpd/alias -n ipaCert -a  | sed -rn '/^-----BEGIN CERTIFICATE-----$/{:1;n;/^-----END CERTIFICATE-----$/b2;H;b1};:2;${x;s/\s//g;p}'  > /tmp/ipaCert.pem

  • Using the ldapmodify command, replace the previous serial number in the description attribute with the new one and add the updated agent certificate blob with the contents of /tmp/ipaCert.pem.

  • The change will look something like the below example (do NOT include brackets around the new values): 

# ldapmodify -x -h localhost -p 7389 -D 'cn=directory manager' -w password
dn: uid=ipara,ou=people,o=ipaca
changetype: modify
add: usercertificate
usercertificate:: <<PASTE NEW AGENT CERTIFICATE BLOB HERE>>
-
replace: description
description: 2;<<INSERT NEW SERIAL NUMBER HERE>>;CN=Certificate Authority,O=EXAMPLE.COM;CN=IPA RA,O=EXAMPLE.COM
       
^D to save and exit

# service httpd restart

Step 9: Renew Directory Server and Apache SSL server certificates

  • Next we need to renew the two Directory Server server certificates and the Apache server certificate:
# ipa-getcert list | grep "Request ID"
  • For each of the three Request IDs run something like this:
# ipa-getcert resubmit -i [Request ID]

Step 10: Return the system clock to the correct time and date and confirm it is now current

  • To return to the present time:
# service ntpd start
# date                           #(confirm it is now current)

Step 11: Return services to production

  • Restart the world:
# service ipa restart
  • Restart certmonger:
# service certmonger restart

Step 12: Test the IPA Admin user

  • Log into IPA as an administrative user:
# kinit admin

Step 13: Test IPA administrative functions

  • Finally, to make sure that IPA administrative functions work correctly, run a few commands to ensure expected data is returned and that no errors are reported:
# ipa cert-show 1
# ipa user-show <userid>
# ipa service-find
# ipa sudorule-find
# ipa hostgroup-find

Root Cause

Several certificates tracked by certmonger are shown with a status of CA_UNREACHABLE:

Request ID '20111111152846':
        status: CA_UNREACHABLE
        ca-error: Server failed request, will retry: -504 (libcurl failed to execute the HTTP POST transaction.  Peer certificate cannot be authenticated with known CA certificates).
        stuck: yes
        key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
        certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
        CA: IPA
        issuer: CN=Certificate Authority,O=EXAMPLE.COM
        subject: CN=ipa.example.com,O=EXAMPLE.COM
        expires: 2013-11-11 15:28:46 UTC
        eku: id-kp-serverAuth,id-kp-clientAuth
        pre-save command: 
        post-save command: /usr/lib64/ipa/certmonger/restart_httpd
        track: yes
        auto-renew: yes
SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.