Repos will not sync to Satellite 6 server -- tasks stuck in Waiting state

Environment

Red Hat Satellite 6.2

Issue

Tasks such as syncing a repo stuck in 'Waiting' state indefinitely

Resolution

1. First perform these steps to clean tasks from Foreman. Read everything very carefully before execution, as some of the query commands will require some slight modification to catch everything.

For example:

# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label ~ *' AFTER='1h' \ STATES='waiting,stopped,paused,pending' VERBOSE=true NOOP=true
# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label ~ *' AFTER='1h' \ STATES='waiting,stopped,paused,pending' VERBOSE=true

2. Install pulp-admin-client -- you can find instructions for this here

3. Next please execute the following to clean tasks from Pulp:

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

STATE=""
for TASK in `pulp-admin -u admin -p $pulpAdminPassword tasks list | egrep '^Task Id:|^State:' | sed -e 's,^Task Id: ,,' -e 's,^State: ,,'`; do
        if [ "$STATE" = "" ]; then
                STATE=$TASK
        else
                if [ $STATE != Successful ] && [ $STATE != Cancelled ] && [ $STATE != Failed ]; then
                        pulp-admin -u admin -p $pulpAdminPassword tasks details --task-id=$TASK
                        pulp-admin -u admin -p $pulpAdminPassword tasks cancel --task-id=$TASK
                fi
                STATE=""
        fi
done

4. Restart pulp and foreman-tasks services with the command:

# for i in pulp_celerybeat pulp_resource_manager pulp_workers foreman-tasks; \ do service $i restart; done

5. Check the health of the Satellite by running

# katello-service status <---- last line of output should be 'Success!'
# hammer ping <---- all services should show 'Status: ok'

6. From the Satellite WebUI, initialize sync of a single repo, e.g. rhel-7-server-rpms -- does this complete?

7. If the above procedure fails, please consider upgrading to Satellite 6.3 or newer.

For more KB articles/solutions related to Red Hat Satellite 6.x Repository Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Repository Issues.

Root Cause

Stuck tasks prevents other tasks from starting due to limited number of available pulp workers

Diagnostic Steps

Observe indefinitely 'Waiting' tasks in Satellite WebUI, foreman task export, or pulp admin task list

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.