[Satellite 6] Candlepin service is down after an upgrade to Satellite 6.9

Environment

• Red Hat Satellite 6.9

Issue

• hammer ping shows that candlepin_events service is down after an upgrade to Satellite 6.9:

candlepin_events:
    Status:          FAIL
    message:         Not running
    Server Response: Duration: 0ms

• satellite-maintain health check reports:

Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [FAIL]
Couldn't connect to the server: undefined method `to_sym' for nil:NilClass
--------------------------------------------------------------------------------

Resolution

• Regenerate the candlepin truststore:

rm -rf /etc/candlepin/certs/truststore && foreman-installer

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

• This issue is being tracked in the following bugzilla: This content is not included.The Artemis client certificate is not updated in truststore if it changes

Diagnostic Steps

• Following errors are logged to candlepin error log:

2021-05-10 00:00:05,925 [thread=Thread-8 (activemq-netty-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222208: SSL handshake failed for client from /127.0.0.1:47148: sun.security.validator.ValidatorException: Netscape cert type does not permit use for SSL client.
2021-05-10 00:00:15,914 [thread=Thread-4 (ActiveMQ-scheduled-threads)] [=, org=, csid=] ERROR org.apache.activemq.artemis.core.server - AMQ224088: Timeout (10 seconds) on acceptor "stomp" during protocol handshake with /127.0.0.1:47148 has occurred.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.