[Satellite 6.2] ERROR: There are 0 pulp_resource_manager processes running.

Environment

• Red Hat Satellite 6.2.x

Issue

• All the tasks in the Red Hat Satellite server goes in pending state.
• Red Hat Satellite 6.2 throws below error in /var/log/messages:

ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.

• pulp_resource_manager service gives following error message

#Katello-service status
--pulp_resource_manager.service 
nov 10 13:32:09 XXXX-sat pulp[20238]: celery.worker:ERROR: (20238-19200) OSError: [Errno 13] Permission denied: '/var/cache/pulp/resource_manager@XXXX-sat'

Resolution

• Check disk space is not 100% full. This could stop new pulp worker to start.
• If selinux is enabled then set the correct selinux context in the pulp folder (/var/cache/pulp/) .
• Run the following command and then check if the pulp services are started successfully.

# df -h
# restorecon -RFvv /var/cache/pulp/ 
# katello-service restart
# cd /var/cache/pulp/
# ls -lZart

• Once this is done, check the output of katello service status. Check the permission and context of pulp files (var/cache/pulp/). Here, resource_manager and reserved_resource_worker file should be created successfully.
• Another possibility is to turn-off(disable) selinux enforce mode and restart katello services

# setenforce 0
# katello-service restart

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

• As per error, there is 0 pulp_resource_manager or worker running; therefore, all the tasks in the satellite server will go for pending state. There is an issue with the wrong selinux context/permission causing pulp_resource_manager.service in INACTIVE state.
• Wrong SE Linux context or permission on pulp resource manager and worker files.
• Basically, permission/selinux context should look like below. If it is not then it should be changed.

#cd /var/cache/pulp/
#ls -lZart
drwxr-xr-x. apache apache system_u:object_r:pulp_var_cache_t:s0 resource_manager@XXXXXXX
drwxr-xr-x. apache apache system_u:object_r:pulp_var_cache_t:s0 reserved_resource_worker-0@XXXXXXX 
drwxr-xr-x. apache apache system_u:object_r:pulp_var_cache_t:s0 reserved_resource_worker-1@XXXXXXX

• Sometimes resource-manager or worker files might be missing in this directory because pulp_resource_manager service is in INACTIVE status. It will be created after setting proper selinux context to the pulp folder.

system_u:object_r:pulp_var_cache_t:s0

Diagnostic Steps

• Satellite 6.2 throws below error in /var/log/messages

Nov 21 08:36:25 ls001e01-00-sat qpidd: 2016-11-21 08:36:25 [System] error Error reading socket: Success(0)
Nov 21 08:36:25 ls001e01-00-sat qpidd[13349]: 2016-11-21 08:36:25 [System] error Error reading socket: Success(0)
Nov 21 08:37:17 ls001e01-00-sat pulp: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.

• Katello service error messages

#Katello-service status

--qpidd.service 
nov 15 09:22:09 XXXX-sat qpidd[19883]: 2016-11-15 09:22:09 [System] error Error reading socket: Success(0)
nov 15 09:22:09 XXXX-sat qpidd[19883]: 2016-11-15 09:22:09 [System] error Error reading socket: Success(0)9:18:08 [System] error Error reading socket: Success(0)

--pulp_celerybeat.service
nov 15 09:21:26 XXXX-sat pulp[20219]: pulp.server.async.scheduler:ERROR: There are 0 pulp_resource_manager processes running. Pulp will not operate correctly without at least one pulp_resource_mananger process running.
nov 15 09:22:56 XXXX-sat pulp[20219]: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)

--pulp_resource_manager.service 
Active: inactive (dead) since gio 2016-11-10 13:32:09 CET; 4 days ago     <<<---INACTIVE STATUS for service pulp_resource_manager.service 
:::::
nov 10 13:32:09 XXXX-sat pulp[20238]: celery.worker:ERROR: (20238-19200)     os.mkdir(working_dir_root)
nov 10 13:32:09 XXXX-sat pulp[20238]: celery.worker:ERROR: (20238-19200) OSError: [Errno 13] Permission denied: '/var/cache/pulp/resource_manager@XXXX-sat'

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.