Pulp tasks hangs with "Waiting for Pulp to start the task"

Environment

• Red Hat Satellite 6 (any version)

Issue

• Used foreman-rake console to destroy Pulp sync tasks and now the new syncs are not starting

• Skipped the tasks related to syncs and now the new syncs are not starting

Resolution

Procedure for Satellite 6.10 or newer (pulp-3)

• Create a script somewhere on the filesystem where you will remember the location and paste this into the script:

#!/bin/bash
certs="--cacert /etc/pki/katello/certs/katello-server-ca.crt --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key"
hname=$(hostname -f)

while :  # we will break if no tasks pending
do
  hrefs=$(curl -s $certs https://${hname}/pulp/api/v3/tasks/?state__in=running,waiting | json_reformat | grep pulp_href | cut -d\" -f4)
  if [ "${hrefs}" == "" ]; then
    break
  fi
  for href in $hrefs; do
    echo "$(date): canceling task $href"
    curl -s $certs -X PATCH -H "Content-Type: application/json" -d "{\"state\": \"canceled\"}" https://${hname}/${href}
    echo; echo
  done
  sleep 1
done

• Now make the script executable

# chmod +x <pathtoscript>

• Now run the script

# /bin/bash <pathtoscript>

Procedure for Satellite 6.9 or older (pulp-2)

• Install pulp-admin and configure it with this article

• Create a script somewhere on the filesystem where you will remember the location and paste this into the script

#!/bin/bash
STATE=""
for TASK in `pulp-admin tasks list | egrep '^Task Id:|^State:' | sed -e 's,^Task Id: ,,' -e 's,^State: ,,'`; do
        if [ "$STATE" = "" ]; then
                STATE=$TASK
        else
                if [ $STATE != Successful ] && [ $STATE != Cancelled ] && [ $STATE != Failed ]; then
                        pulp-admin tasks details --task-id=$TASK
                        pulp-admin tasks cancel --task-id=$TASK
                fi
                STATE=""
        fi
done

• Now make the script executable

# chmod +x <pathtoscript>

• Now run the script

# /bin/bash <pathtoscript>

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 3.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 3.0-related Issues

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

• When destroying tasks with foreman-rake console or skipping tasks, pulp does not get the api call to cancel the tasks so they are still in a paused/waiting state.

Diagnostic Steps

• You can check if there are tasks currently in a paused/waiting state with pulp-admin by doing the following

# pulp-3 / Satellite 6.10 or newer
certs="--cacert /etc/pki/katello/certs/katello-server-ca.crt --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key"
hname=$(hostname -f)
curl -s $certs https://${hname}/pulp/api/v3/tasks/?state__in=running,waiting | json_reformat | grep -m1 count

# pulp-2 / Satellite 6.9 or older
pulp-admin tasks list | grep -i waiting | wc -l

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.