After upgrading to Satellite 6.3, candlepin is in suspended mode

Environment

• Red Hat Satellite 6.3
  • upgraded from an older version

Issue

• upgrading Satellite to 6.3
• just since then, candlepin complains it is in suspended mode with an SSL hostname verification failed: Expected : localhost Found in cert : [satellite.example.com] error.

Resolution

Move to a backup nssdb certificates for qpidd that were insufficiently updated, and let installer to regenerate them:

mv /etc/pki/katello/nssdb{,.backup}
satellite-installer

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

An upgrade to Satellite 6.3 updates configuration of qpidd to let it listen only on localhost and being contacted via localhost URI only. That implies its SSL certificates must contain alternate DNS of localhost, otherwise hostname verification check during SSL handshake would fail.

Due to an unknown bug, this SSL certificates update does not fully happen. The cert.file /etc/pki/katello/certs/satellite.example.com-qpid-broker.crt is updated properly, but broker certificate within /etc/pki/katello/nssdb (being used by qpidd) does not. Removing this whole NSS database will force subsequent installer to regenerate them, now with proper content.

Diagnostic Steps

/var/log/candlepin/error.log contains:

2018-08-07 13:18:29,295 [thread=IoReceiver - localhost/127.0.0.1:5671] [=, org=, csid=] ERROR org.apache.qpid.transport.Connection - exception: SSL hostname verification failed. Expected : localhost Found in cert : [satellite.example.com]
org.apache.qpid.transport.ConnectionException: SSL hostname verification failed. Expected : localhost Found in cert : [satellite.example.com]
        at org.apache.qpid.transport.Connection.exception(Connection.java:561) [qpid-common-0.32.jar:0.32]
        at org.apache.qpid.transport.network.Assembler.exception(Assembler.java:107) [qpid-common-0.32.jar:0.32]
        at org.apache.qpid.transport.network.InputHandler.exception(InputHandler.java:209) [qpid-common-0.32.jar:0.32]
        at org.apache.qpid.transport.network.security.ssl.SSLReceiver.exception(SSLReceiver.java:72) [qpid-common-0.32.jar:0.32]
        at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:217) [qpid-common-0.32.jar:0.32]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: org.apache.qpid.transport.TransportException: SSL hostname verification failed. Expected : localhost Found in cert : [satellite.example.com]
        at org.apache.qpid.transport.network.security.ssl.SSLUtil.verifyHostname(SSLUtil.java:141) ~[qpid-common-0.32.jar:0.32]
        at org.apache.qpid.transport.network.security.ssl.SSLUtil.verifyHostname(SSLUtil.java:72) ~[qpid-common-0.32.jar:0.32]
        at org.apache.qpid.transport.network.security.ssl.SSLReceiver.received(SSLReceiver.java:171) [qpid-common-0.32.jar:0.32]
        at org.apache.qpid.transport.network.security.ssl.SSLReceiver.received(SSLReceiver.java:36) [qpid-common-0.32.jar:0.32]
        at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161) [qpid-common-0.32.jar:0.32]
        ... 1 common frames omitted

optional checks show the localhost is present in `` :

openssl x509 -in /etc/pki/katello/certs/satellite.example.com-qpid-broker.crt -text
..
            X509v3 Subject Alternative Name: 
                DNS:satellite.example.com, DNS:localhost
..

but not in NSS database:

certutil -L -d /etc/pki/katello/nssdb -n broker
..

            Name: Certificate Subject Alt Name
            DNS name: "satellite.example.com"
..

(see the missing DNS name: "localhost" there)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.