pulp-3 workers remain idle despite there is a pending task

Environment

• Red Hat Satellite 6.10 onward.

Issue

• Random big bulk action (many CVs deletions, Capsule sync, a big Sync Plan of small repos,..) gets stuck on a pending pulp task forever, usually on the latest task. Deeper inspection shows all pulp workers are idle, but the task is still pending.

Resolution

• The issue has been solved in Satellite 6.11 via RHSA-2022:5498 and back ported to 6.10.6 via RHBA-2022:4850.

• If upgrade is not an option, two independent workarounds are provided below:

• Workaround 1: invoke any action that triggers a new pulp task. Both the new task(s) and the stuck one will be managed by the workers.

• Workaround 2: notify the workers via postgres command:

  su - postgres -c "psql pulpcore -c \"NOTIFY pulp_worker_wakeup;\""
  

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 3.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 3.0-related Issues

Root Cause

• There is a race condition leading to the latest active worker assumed to execute the latest (bunch of) pending task(s) to skip the due work and to go to an idle state. A wake up from that idle state is either any new task created, or a direct notification via postgres.

Diagnostic Steps

• Pulp tasks show all except one task (or a few similar one tasks, all manipulating with a same object) are finished and the remaining as waiting.

• All pulp workers are idle, not consuming CPU, if inspecting their backtrace then it is expected to be like:

Traceback (most recent call first):
 <built-in method select of module object at remote 0x7f8260cc8ea8>
 File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 190, in sleep
   [self.sentinel, connection.connection], [], [], self.heartbeat_period
 File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 297, in run_forever
   self.sleep()
 File "/usr/lib/python3.6/site-packages/pulpcore/tasking/entrypoint.py", line 44, in worker
   NewPulpWorker().run_forever()
 File "/usr/lib/python3.6/site-packages/click/core.py", line 610, in invoke
   return callback(*args, **kwargs)
 File "/usr/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
   return ctx.invoke(self.callback, **ctx.params)
 File "/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
   rv = self.invoke(ctx)
 File "/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
   return self.main(*args, **kwargs)
 File "/usr/bin/pulpcore-worker", line 11, in <module>
   load_entry_point('pulpcore==3.14.14', 'console_scripts', 'pulpcore-worker')()

• Notifying workers via the NOTIFY pulp_worker_wakeup; command triggers the task execution start.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.