hammer ping shows candlepin FAIL with error Message: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired) on Red Hat Satellite

Environment

• Red Hat Satellite 6

Issue

• The hammer ping command shows candlepin with status FAIL even though the Candlepin certificate is not expired.

Resolution

1. Follow the steps in the Diagnostic Steps section to confirm that this solution fits the issue at hand. If not, do not proceed with the next steps.

2. Take a backup/snapshot of the Red Hat Satellite server.

3. Run the following commands on the Satellite server:

# mv /root/ssl-build/localhost /root/ssl-build/localhost_old
# rm -f ~/candlepin_cert_bak/*
# mv /etc/candlepin/certs/truststore ~/candlepin_cert_bak/
# mv /etc/candlepin/certs/keystore  ~/candlepin_cert_bak/
# mv /etc/candlepin/certs/candlepin-ca.crt ~/candlepin_cert_bak/
# mv /etc/candlepin/certs/candlepin-ca.key ~/candlepin_cert_bak/
# rm -rf /etc/pki/katello/nssdb
# satellite-installer
# hammer ping

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

For more KB articles/solutions related to Red Hat Satellite 6.x SSL Certificates Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x SSL Certificates Issues

Root Cause

• The candlepin certificate got corrupted which is causing issues while running the hammer ping.

Diagnostic Steps

1. Examine the output of the hammer ping command:

# hammer ping
.
.
candlepin:        
    Status:          FAIL
    Server Response: Message: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)
candlepin_auth:   
    Status:          FAIL
    Server Response: Message: Katello::Errors::CandlepinNotRunning
candlepin_events: 
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 0ms
.
.

2. Verify that the Candlepin certificate is not expired:

# openssl crl2pkcs7 -nocrl -certfile /etc/candlepin/certs/candlepin-ca.crt  | openssl pkcs7 -text -print_certs |egrep '(Issuer:|Subject:|CA:|DNS:|Digital|Not Before|Not After|keyid|serial:|TLS)'
        Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=satellite.example.com
            Not Before: Feb 12 10:45:33 2019 GMT
            Not After : Jan 18 10:45:33 2038 GMT
        Subject: C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=satellite.example.com
                CA:TRUE
                Digital Signature, Key Encipherment, Certificate Sign, CRL Sign
                TLS Web Server Authentication, TLS Web Client Authentication
                keyid:AB:D0:A9:78:41:5F:4A:05:3E:E1:33:11:63:42:6E:E3:EB:88:27:85
                serial:F8:44:9D:FC:09:08:6F:40

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.