Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Upgrade Issues

• [Satellite6] Upgrade of Satellite Server results in a FAIL for pulp and pulp_auth

Pulp Services and/or Processes Issues

• Red Hat Satellite 6 pulp and pulp_auth are not starting.
• [Satellite 6] pulp services are down because some of the pulp plugins failed to initialize
• [Satellite 6.2] ERROR: There are 0 pulp_resource_manager processes running.
• Hammer ping failing or pulp and pulp_auth failing
• Rebooting the Satellite6 server causes pulp services won't start
• [Satellite6] pulp remove orphans task runs many hours
• [Satellite 6] There are 0 pulp_celerybeat processes running
• [Satellite6] various pulp tasks are pending
• Pulp tasks are not processed in Red Hat Satellite and it hangs in the running state with error E11000 duplicate key error index: pulp_database.task_status.$task_id_-1 dup key
• [Satellite6] Various tasks are pending on "waiting for Pulp to start the task"
• Mongod fails after start in Satellite6 with "invalid file index requested" error
• Satellite6 or Capsule stops processing pulp tasks
• Capsule sync stuck close to end forever

Pulp Filesystem/Storage Issues

• Pulp broken symlinks are found in "/var/lib/pulp" after upgrading Red Hat Satellite 6
• The /var/lib/pulp file system is growing faster in Red Hat Satellite 6
• On Red Hat Satellite 6.3 '/var/log/messages' shows errors: 'pulp.streamer.server:ERROR: No catalog entry found. path=/var/lib/pulp/content/units/rpm/'
• On Red Hat Satellite 6, /var/lib/pulp/content consumes much more space than expected
• Removing a repository does not delete its content in /var/lib/pulp on Red Hat Satellite Server.

Pulp workers Issues

• [Satellite6] Some pulp workers are missing after Satellite reboot
• [Satellite6] WebUI shows more Capsule pulp workers than configured
• [Satellite6] Synchronizing Atomic OSTRee repo fails, pulp worker dies
• This content is not included.[Satellite6] pulp tasks stuck forever, worker processes are idle despite there are pending tasks
• Pulp celery workers cannot connect to qpid

Other Pulp Issues

• Pulp tasks hangs with "Waiting for Pulp to start the task"
• Capsule synchronization seems stuck at 10% or 11% for a long time
• [Satellite 6.2] pulp raises E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1
• [Satellite 6.2] pulp raises "DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_erratum"
• [Satellite 6.2] pulp raises "DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_rpm"
• [Satellite 6.2] pulp raises DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_docker_tag.$name_1_repo_id_1
• [Satellite6] all pulp tasks are failing, wsgi:pulp process segfaults / segfault at 10 ip ... error 4 in libpython2.7.so.1.0
• Forcefully republishing pulp repository makes no real effect
• Can't run foreman-rake katello:correct_repositories or correct_puppet_environments to create missing repositories in Pulp
• [Satellite 6.7] regenerate applicability is empty for random Content Hosts
• [Satellite6] What data to provide for pulp repo sync issues?
• [Satellite6] Task "waiting for Pulp to start the task" is hung forever
• [Satellite6] katello:correct_repositories rake script raises PG::UniqueViolation when creating missing pulp repo for custom product
• /var/log/messages spammed by many mongod.27017 logs on Satellite6 or Capsule
• Sync tasks keeps running in Red Hat Satellite v6

Pulp_Streamer service Issues

• Satellite Capsule pulp_streamer.service entered failed state
• Unable to start pulp_streamer service after fresh installation of Red Hat Satellite 6.6

How To

• How to obtain the size of repositories in Red Hat Satellite
• Red Hat Satellite 6 : How to deploy and use pulp-admin?
• How to change the number of pulp workers on Red Hat Satellite 6?
• How to cancel all pulp tasks on a Capsule or Satellite?
• How to delete orphaned content in /var/lib/pulp on Capsule?
• [Satellite6] How to mimic fetching a package or metadata for a repository on Satellite or Capsule?
• [Satellite6] how to mimic from command line repository sync requests that are failing?
• How to restore missing or corrupted docker_blob file in Satellite6?
• [Satellite6] How to dump queries to pulp or candlepin from katello/foreman?
• How to get summary of pulp tasks from katello task export?
• Directory structure of /var/lib/pulp in Satellite6 or Capsule
• [Satellite6] How to get rid of pulp-admin wrapping output to 80 characters?
• How to cancel Pulp related foreman tasks in batch
• [Satellite6] How to fix a broken repository on Capsule?
• [Satellite6] How to regenerate applicability of a repository?
• Content Host missing a package metadata - how to troubleshoot or potentially fix it?
• How to restore just mongo database for Satellite6?
• [Satellite6] How to generate postgres+mongo tables to understand databases content?
• How to reset the pulp password in Red Hat Satellite 6?
• How to move the content of /var/lib/pulp into a new logical volume?

Q & A

• What is the recommended size of /var/cache/pulp on Red Hat Satellite server?
• Is it safe to symlink pulp cache to /var/lib/pulp for Satellite6 or its Capsule?