Error with STOMP client causing 'candlepin_events' to enter a failed state

Solution Verified - Updated

Environment

  • Red Hat Satellite 6

Issue

The candlepin_events process is currently in a failed state.

candlepin_events: 
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 0ms

The below error is being observed on /var/log/candlepin/error.log:

2021-03-17 13:38:50,666 [thread=Thread-8 (activemq-netty-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.protocol.stomp - AMQ332069: Sent ERROR frame to STOMP client /127.0.0.1:36718: Security Error occurred: User name [null] or password is invalid

2021-03-17 13:39:05,842 [thread=Thread-9 (activemq-netty-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222216: Security problem while authenticating: AMQ229031: Unable to validate user from /127.0.0.1:36726. Username: null; SSL certificate subject DN: CN=localhost, OU=SomeOrgUnit, O=candlepin, ST=North Carolina, C=US

2021-03-17 13:39:05,842 [thread=Thread-9 (activemq-netty-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.protocol.stomp - AMQ332069: Sent ERROR frame to STOMP client /127.0.0.1:36726: Security Error occurred: User name [null] or password is invalid

2021-03-17 13:39:21,033 [thread=Thread-10 (activemq-netty-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222216: Security problem while authenticating: AMQ229031: Unable to validate user from /127.0.0.1:36730. Username: null; SSL certificate subject DN: CN=localhost, OU=SomeOrgUnit, O=candlepin, ST=North Carolina, C=US

  • # satellite-maintain service restart fails with:

    Check whether all services are running using the ping call:           [FAIL]
Couldn't connect to the server: undefined method `to_sym' for nil:NilClass

Resolution

# satellite-maintain service stop

Next, please edit the /etc/tomcat/cert-users.properties file on your Satellite. The file currently has the contents below:

katelloUser=CN=localhost, OU=SomeOrgUnit, O=candlepin, ST=North Carolina, C=US

Note: These are not in line with what the logs report Satellite is using as a client certificate to authenticate against Candlepin's STOMP acceptor.
From the /var/log/candlepin/error.log the properties can be seen that candlepin is expecting to use:

2021-11-08 12:29:13,042 [thread=Thread-43 (activemq-netty-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222216: Security problem while authenticating: AMQ229031: Unable to validate user from /127.0.0.1:51522. Username: null; SSL certificate subject DN: CN=satellite.example.com, OU=SomeOrgUnit, O=candlepin, ST=North Carolina, C=US

You can use the following command to grep out the exact string needed for this file from this error:

# echo "katelloUser=$(grep -o DN.* /var/log/candlepin/error.log | sort -u | cut -d' ' -f2-7)"

Example:

[root@satellite ~]# echo "katelloUser=$(grep -o DN.* /var/log/candlepin/error.log | sort -u | cut -d' ' -f2-7)"
katelloUser=CN=satellite.example.com, OU=SomeOrgUnit, O=candlepin, ST=North Carolina, C=US

[root@satellite ~]# echo "katelloUser=$(grep -o DN.* /var/log/candlepin/error.log | sort -u | cut -d' ' -f2-7)" > /etc/tomcat/cert-users.properties

[root@satellite ~]# cat /etc/tomcat/cert-users.properties
katelloUser=CN=satellite.example.com, OU=SomeOrgUnit, O=candlepin, ST=North Carolina, C=US

Next, please start Satellite services and run hammer ping:

# satellite-maintain service start
# hammer ping

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

The wrong katelloUser value was probably put in place during some earlier install of Satellite on this host, when Satellite's OU and ST and C attributes were customized for its internal certificate authority. This is usually accomplished by using the arguments below with the satellite-installer command:

`--certs-org-unit KVIKT  --certs-state Trondelag  --certs-country NO`

After running satellite-installer with the arguments below, you probably reset these variables so they adopted their default values of SomeOrgUnit, North Carolina, and US, respectively.

Additionally, if the Satellite was configured with custom certificates at some point, the default values for this could still be saved in this file from the new certificates being used on the Satellite.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.