[Satellite6] Candlepin stays in suspended mode, tomcat restart does not help

Solution Verified - Updated

Environment

Red Hat Satellite 6

Issue

  • many actions (subscription related) fail with error that candlepin is in suspended mode
  • candlepin logs Candlepin is entering suspend mode for the following reason: QPID_DOWN even after a restart, despite qpidd is running and providing service

Resolution

Ensure IPv4 network works fine. If so, restart qpidd service:

systemctl restart qpidd.service

To speed up recovery of candlepin, it is recommended to restart tomcat running candlepin app as well:

systemctl restart tomcat.service

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

  • qpidd service was being started at a time when IPv4 network was down (and IPv6 was working only)
  • qpidd then started to listen on IPv6 socket only
  • candlepin is however configured / able to connect to qpidd over 127.0.0.1 only
  • so any candlepin's attempt to (re)connect to qpidd - even after candlepin/tomcat restart - fails due to closed socket 127.0.0.1::5671
  • restarting qpidd with IPv4 working will resolve the issue

Diagnostic Steps

  • katello_events_queue queue in qpidd is empty, no issue with sending messages there
  • event exchange is present in qpidd as well
  • /var/log/candlepin/error.log contains after each tomcat restart:
2019-02-23 16:33:14,199 [thread=localhost-startStop-1] [=, org=, csid=] WARN  org.candlepin.controller.ModeManagerImpl - Candlepin is entering suspend mode for the following reason: QPID_DOWN
  • netstat shows qpidd listening on port 5671 BUT only on IPv6:
# netstat -anp | grep LISTEN | grep qpidd
tcp6       0      0 ::1:5671                :::*                    LISTEN      5022/qpidd
#
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.