[Satellite 6] Task to publish new version of content view stuck on 'waiting for Pulp to finish the task' and cannot be deleted

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.3 or newer

Issue

We are publishing a new version of a content view on our Red Hat Satellite server. About 20 hours passed since the task to publish the new version of the content view was started, and the task is still stuck at 6%.
We need to delete this task and start over with a new one. We tried to delete the task by restarting pulp services, and it is still there.

Resolution

  • Delete the Actions::Katello::Repository::CapsuleGenerateAndSync that are in reported in paused:error state using the command:
# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label ~ *' VERBOSE=true STATES='paused'
  • Delete the Actions::Katello::ContentView::Publish task that was blocking production, using the following command:
# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = Actions::Katello::ContentView::Publish' STATES='running,pending,stopped' VERBOSE=true

  • If it is not already deployed on Satellite server, deploy pulp-admin by following the steps described in this KCS.

  • Create a script somewhere on the filesystem where you will remember the location and paste this into the script:

#!/bin/bash

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

STATE=""
for TASK in `pulp-admin -u admin -p $pulpAdminPassword tasks list | egrep '^Task Id:|^State:' | sed -e 's,^Task Id: ,,' -e 's,^State: ,,'`; do
        if [ "$STATE" = "" ]; then
                STATE=$TASK
        else
                if [ $STATE != Successful ] && [ $STATE != Cancelled ] && [ $STATE != Failed ]; then
                        pulp-admin -u admin -p $pulpAdminPassword tasks details --task-id=$TASK
                        pulp-admin -u admin -p $pulpAdminPassword tasks cancel --task-id=$TASK
                fi
                STATE=""
        fi
done
  • Now make the script executable:
# chmod +x <pathtoscript>
  • Now run the script:
# /bin/bash <pathtoscript>
  • Restart pulp services:
# for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i restart; done
  • In Satellite server web UI, perform the following steps:
    1. Delete the version of the content view for which the publish task was deleted after it got stuck.
    2. Publish a new version of the content view, and promote it to the desired environment(s).

For more KB articles/solutions related to Red Hat Satellite 6.x Content View Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Content View Issues

Diagnostic Steps

  • Output from the following command on the Satellite server indicates some several tasks in paused:error state:
# sudo su - postgres -c "psql -d foreman -c 'select label,count(label),state,result from foreman_tasks_tasks where state <> '\''stopped'\'' group by label,state,result ORDER BY label;'"
                         label                         | count |   state   | result
-------------------------------------------------------+-------+-----------+---------
 Actions::Candlepin::ListenOnCandlepinEvents           |     1 | running   | pending
 Actions::Insights::EmailPoller                        |     1 | scheduled | pending
 Actions::Katello::CapsuleContent::Sync                |     1 | paused    | error
 Actions::Katello::CapsuleContent::Sync                |   160 | running   | pending
 Actions::Katello::CapsuleContent::Sync                |    25 | planning  | pending
 Actions::Katello::ContentView::CapsuleGenerateAndSync |    14 | paused    | error
 Actions::Katello::ContentView::CapsuleGenerateAndSync |    36 | running   | pending
 Actions::Katello::ContentView::Publish                |     1 | running   | pending
 Actions::Katello::ContentView::Publish                |     1 | paused    | error
 Actions::Katello::EventQueue::Monitor                 |     1 | running   | pending
 Actions::Katello::Host::Destroy                       |     1 | paused    | error
 Actions::Katello::Host::GenerateApplicability         |    21 | paused    | error
 Actions::Katello::Host::Package::Update               |     1 | running   | pending
 Actions::Katello::Host::Update                        |     1 | running   | pending
 Actions::Katello::Host::UploadPackageProfile          |     5 | running   | pending
 Actions::Katello::Organization::ManifestRefresh       |     1 | running   | pending
 Actions::Katello::Repository::CapsuleGenerateAndSync  |    42 | paused    | error
 Actions::Katello::Repository::CapsuleGenerateAndSync  |     8 | running   | pending
 Actions::Katello::Repository::MetadataGenerate        |     3 | paused    | error
 Actions::Katello::Repository::ScheduledSync           |    79 | running   | pending
 Actions::Katello::Repository::ScheduledSync           |     1 | paused    | error
 Actions::Katello::RepositorySet::ScanCdn              |     1 | running   | success
(22 rows)

  • Under Monitor > Tasks in Satellite web UI, several Actions::Katello::Repository::ScheduledSync tasks stuck at this stage:
    Actions::Pulp::Repository::RegenerateApplicability (suspended)

  • Under Monitor > Tasks in Satellite web UI, several Actions::Katello::CapsuleContent::Sync tasks are in running:pending state with Actions::Pulp::Consumer::SyncCapsule (waiting for Pulp to finish the task) whose output looks like the following:

---
pulp_tasks:
- exception: 
  task_type: pulp.server.managers.repo.sync.sync
  _href: "/pulp/api/v2/tasks/8cc74bc4-8aac-4f87-add9-3a66928f8347/"
  task_id: 8cc74bc4-8aac-4f87-add9-3a66928f8347
  tags:
  - pulp:repository:1-ccv-fadaba-Integration-DLZ-IT-EPEL_7-EPEL_7_Repo
  - pulp:action:sync
  finish_time: 
  _ns: task_status
  start_time: '2019-09-17T12:43:53Z'
  traceback: 
  spawned_tasks: []
  progress_report:
    yum_importer:
      content:
        items_total: 2881
        state: IN_PROGRESS
        error_details:
        - url: https://example.satellite.com/pulp/repos/DLZ-IT/Integration/ccv-fadaba/custom/EPEL_7/EPEL_7_Repo/Packages/f/fuse-encfs-1.9.4-1.el7.x86_64.rpm
          errors:
          - Not Found
        - url: https://example.satellite.com/pulp/repos/DLZ-IT/Integration/ccv-fadaba/custom/EPEL_7/EPEL_7_Repo/Packages/l/lizardfs-adm-3.11.0-2.el7.x86_64.rpm
          errors:
          - Not Found
        - url: https://example.satellite.com/pulp/repos/DLZ-IT/Integration/ccv-fadaba/custom/EPEL_7/EPEL_7_Repo/Packages/l/libabigail-devel-1.1-1.el7.x86_64.rpm
          errors:
          - Not Found
        - url: https://example.satellite.com/pulp/repos/DLZ-IT/Integration/ccv-fadaba/custom/EPEL_7/EPEL_7_Repo/Packages/g/GraphicsMagick-perl-1.3.26-1.el7.x86_64.rpm
          errors:
          - Not Found
        - url: https://example.satellite.com/pulp/repos/DLZ-IT/Integration/ccv-fadaba/custom/EPEL_7/EPEL_7_Repo/Packages/p/python2-certbot-nginx-0.14.1-1.el7.noarch.rpm
          errors:
          - Not Found
.
.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.