[Satellite6] various pulp tasks are pending

Solution Verified - Updated

Environment

  • Red Hat Satellite 6

Issue

  • various pulp tasks are pending, like:
    • sync from CDN
    • publish a new version of a Content View
    • synchronize Capsule

Resolution

Ensure pulp workers are running and subscribed to their queues:

# ps aux | grep pulp | grep -e worker -e manager
apache   16592  2.8  0.7 668028 61984 ?        Ssl  12:04   0:00 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30
apache   16623 13.0  0.8 668812 65456 ?        Ssl  12:04   0:04 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid --heartbeat-interval=30
apache   16625 14.0  0.8 669204 65692 ?        Ssl  12:04   0:04 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-1.pid --heartbeat-interval=30
apache   16627 14.4  0.8 668656 65632 ?        Ssl  12:04   0:04 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-2.pid --heartbeat-interval=30
apache   16631 13.7  0.7 668816 63604 ?        Ssl  12:04   0:04 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-3.pid --heartbeat-interval=30
apache   16717  8.5  0.7 889988 58600 ?        Sl   12:04   0:02 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-2@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-2.pid --heartbeat-interval=30
apache   16718  8.0  0.7 816256 56532 ?        Sl   12:04   0:02 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-1@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-1.pid --heartbeat-interval=30
apache   16721  7.1  0.7 1029112 58584 ?       Sl   12:04   0:02 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-0@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-0.pid --heartbeat-interval=30
apache   16722  0.0  0.6 580976 52068 ?        S    12:04   0:00 /usr/bin/python /usr/bin/celery worker -A pulp.server.async.app -n resource_manager@%h -Q resource_manager -c 1 --events --umask 18 --pidfile=/var/run/pulp/resource_manager.pid --heartbeat-interval=30
apache   16727  7.6  0.7 816108 56548 ?        Sl   12:04   0:02 /usr/bin/python /usr/bin/celery worker -n reserved_resource_worker-3@%h -A pulp.server.async.app -c 1 --events --umask 18 --pidfile=/var/run/pulp/reserved_resource_worker-3.pid --heartbeat-interval=30
#

(there should be pair of processes with resource_manager.pid pidfile and several pairs of processes with reserved_resource_worker-*.pid pidfiles)

Ensure the processes are subscribed to their queues - the queues should have exactly one consumer (2nd column from the right):

# qpid-stat -q --ssl-certificate=/etc/pki/katello/qpid_client_striped.crt -b amqps://localhost:5671 | grep resource | grep -v pidbox
  reserved_resource_worker-0@satellite.example.com.dq             Y    Y         0     0      0       0      0        0         1     2
  reserved_resource_worker-1@satellite.example.com.dq             Y    Y          0     0      0       0      0        0         1     2
  reserved_resource_worker-2@satellite.example.com.dq             Y    Y         0     0      0       0      0        0         1     2
  reserved_resource_worker-3@satellite.example.com.dq             Y    Y         0     0      0       0      0        0         1     2
  resource_manager                                                Y              0     1      1       0   1.36k    1.36k        1     2
  resource_manager@satellite.example.com.dq                       Y    Y         0     0      0       0      0        0         1     2
#

If that is not the case, a restart of pulp services should help:

for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i restart; done

If that does not help, please contact Red Hat support and provide sosreport from this system.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

resource_manager queue gets pulp task requests to be executed on the given pulp server. Relevant pulp manager process fetches messages from the queue and assigns the tasks to the most appropriate worker.

reserved_resource_worker-* queues contain tasks scheduled for the given worker. The worker should fetch messages from its queue to start working on the tasks in the messages.

If either process is not subscribed to its queue, it can't fetch messages and process the tasks accordingly.

Diagnostic Steps

qpid-stat -q showing no consumer of either above mentioned queue. This command output is collected by:

  • foreman-debug in qpid_stat_queues file
  • sosreport in sos_commands/foreman/foreman-debug/qpid_stat_queues file
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.