Satellite 6 - How to resolve the error Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found?

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.1
  • Red Hat Satellite 6.2
  • Red Hat Satellite 6.3

Issue

hammer ping results in the following errors:

# hammer ping
[Foreman] Username: admin
[Foreman] Password for admin: 
candlepin:      
    Status:          FAIL
    Server Response: Message: 404 Resource Not Found
candlepin_auth: 
    Status:          FAIL
    Server Response: Message: Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found  (GET /candlepin/status)
pulp:           
    Status:          ok
    Server Response: Duration: 43ms
pulp_auth:      
    Status:          ok
    Server Response: Duration: 21ms
elasticsearch:  
    Status:          ok
    Server Response: Duration: 31ms
foreman_tasks:  
    Status:          ok
    Server Response: Duration: 1ms
  • candlepin error log contains javax.jms.JMSException: Error creating connection: General SSLEngine problem

Note: There can be more reasons other than the above why candlepin returns 404 Resource Not Found in hammer ping.
This KCS describes one use case when after changing certificates, candlepin its truststore was obsolete and caused the General SSLEngine problem.

Resolution

  • The candlepin truststore (list of trust certificate authorities) is obsolete. Move it to a backup to force the installer to regenerate it:

    # mv /etc/candlepin/certs/amqp/candlepin.truststore{,.bak}
# katello-installer

  • In another instance, some of the critical packages such as mongod, ruby*, puppet, candlepin* etc were installed from EPEL repos. Ensure that no critical packages, which were part of Red Hat Satellite 6, RHSCL and RHEL repos were updated from EPEL repos.

  • Disable EPEL repo on Red Hat Satellite 6 server and then remove and re-install all the offending packages
    IMPORTANT: Backup the configuration files of the RPM packages, that would removed and replace them after installing the packages from the Red Hat provided repositories.

      # mv /etc/yum.repos.d/epel* /tmp
  # subscription-manager repos --disable=* --enable=rhel-X-server-rpms --enable=rhel-server-rhscl-X-rpms --enable=rhel-X-server-satellite-6.Y-rpms 
  Note: Here X denotes, Operating System major version i.e. 6 or 7 and Y denotes the Red Hat Satellite minor version 0, 1 or 2 

  # rpm -e <package-name> --nodeps

  # yum install <packagename>

Note: As the above steps are destructive, it is advisable to have a latest backup. If Red Hat Satellite is a VM, then take a snapshot prior to attempting removal of RPMs.

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

/etc/candlepin/certs/amqp/candlepin.truststore contains CA (certificate authority) with proper Subject but wrong ID, that differs from ID in /etc/pki/katello/certs/katello-default-ca-stripped.crt and also authority ID of /etc/candlepin/certs/amqp/candlepin.jks.

You can find more details in the diagnostic steps below when it's wrong.

Here is an example of what it should look like if everything is correct:

# openssl x509 -in /etc/pki/katello/certs/katello-default-ca-stripped.crt -text   | grep "X509v3 Subject Key Identifier" -A1
            X509v3 Subject Key Identifier: 
                08:E1:33:42:1D:47:5D:9E:43:DF:50:F4:DF:74:1F:C0:87:CB:CA:80

# password is value of candlepin.amqp.keystore_password in /etc/candlepin/candlepin.conf
keytool -list -v -keystore /etc/candlepin/certs/amqp/candlepin.truststore | grep KeyIdentifier -A1
Enter keystore password:  

*****************  WARNING WARNING WARNING  *****************
* The integrity of the information stored in your keystore  *
* has NOT been verified!  In order to verify its integrity, *
* you must provide your keystore password.                  *
*****************  WARNING WARNING WARNING  *****************

AuthorityKeyIdentifier [
KeyIdentifier [
0000: 08 E1 33 42 1D 47 5D 9E   43 DF 50 F4 DF 74 1F C0  ..3B.G].C.P..t..
--
SubjectKeyIdentifier [
KeyIdentifier [
0000: 08 E1 33 42 1D 47 5D 9E   43 DF 50 F4 DF 74 1F C0  ..3B.G].C.P..t..
  • Critical packages are updated from EPEL repositories causing instability.

Diagnostic Steps

The following error shows up under candlepin/error.log

2016-01-25 08:44:57,103 [=, org=] ERROR org.candlepin.guice.AMQPBusPubProvider - Unable to instantiate AMQPBusProvider:
javax.jms.JMSException: Error creating connection: General SSLEngine problem
        at org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:128) ~[qpid-client-0.30.redhat-1.jar:0.30.redhat-1]
        at org.candlepin.guice.AMQPBusPubProvider.init(AMQPBusPubProvider.java:84) ~[AMQPBusPubProvider.class:na]

When verifying the certs you see that the subject and CA do not match:

# openssl x509 -in /etc/pki/katello/certs/katello-default-ca-stripped.crt -text   | grep "X509v3 Subject Key Identifier" -A1          
  X509v3 Subject Key Identifier: 
                C3:79:14:C1:17:12:A1:30:43:DF:50:F4:DF:74:1F:C0:87:CB:CA:80

# keytool -list -v -keystore /etc/candlepin/certs/amqp/candlepin.truststore | grep KeyIdentifier -A1
Enter keystore password (press enter):  

*****************  WARNING WARNING WARNING  *****************
* The integrity of the information stored in your keystore  *
* has NOT been verified!  In order to verify its integrity, *
* you must provide your keystore password.                  *
*****************  WARNING WARNING WARNING  *****************

AuthorityKeyIdentifier [
KeyIdentifier [
0000: 08 E1 33 42 1D 47 5D 9E   43 DF 50 F4 DF 74 1F C0  ..3B.G].C.P..t..
--
SubjectKeyIdentifier [
KeyIdentifier [
0000: 08 E1 33 42 1D 47 5D 9E   43 DF 50 F4 DF 74 1F C0  ..3B.G].C.P..t..
  • To find non Red Hat provided RPMs on the Red Hat Satellite server execute the following command
# rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n' | grep -v 'Red Hat, Inc\.' | sort
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.