Pulp celery workers cannot connect to qpid

Solution Verified - Updated

Environment

Satellite 6.x

Issue

In /var/log/messages of Satellite server, it seems that Pulp celery workers are unable to connect to qpid.

Root Cause

Looking at katello-service status or foreman maintain service status it is visible that Satellite services were restarted just some short time ago.

Qpid needs time to reload its journal files and during that time to preserve integrity, it refuses any incoming messages from other Satellite services.

This load time might take few minutes, depending on the number of hosts managed by katello-agent.

Eventually, it can be observed that celery workers were able to connect to qpid:

Apr 29 15:32:03 satellite.example.com pulp[32271]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
Apr 29 15:32:03 satellite.example.com pulp[32271]: celery.worker.consumer:INFO: Connected to qpid://localhost:5671//

If they they are not able to connect at all, it is a different issue. In that case, please contact Red Hat Support.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Diagnostic Steps

/var/log/messages shows: 

Apr 29 15:31:59 satellite.example.com pulp[32271]: celery.worker.consumer:ERROR: (32271-72512) consumer: Cannot connect to qpid://localhost:5671//: [Errno 111] Connection refused.
Apr 29 15:31:59 satellite.example.com pulp[32271]: celery.worker.consumer:ERROR: (32271-72512) Trying again in 4.00 seconds...
Apr 29 15:31:59 satellite.example.com pulp[32271]: celery.worker.consumer:ERROR: (32271-72512)

In katello-service status output we can see qpid was restarted shortly before: 

● qpidd.service - An AMQP message broker daemon.
   Loaded: loaded (/usr/lib/systemd/system/qpidd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/qpidd.service.d
           └─limits.conf
   Active: active (running) since Mon 2019-04-29 15:31:55 EDT; 55s ago
     Docs: man:qpidd(1)
           http://qpid.apache.org/
 Main PID: 32225 (qpidd)
   CGroup: /system.slice/qpidd.service
           └─32225 /usr/sbin/qpidd --config /etc/qpid/qpidd.conf
SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.