Pulp broken symlinks are found in "/var/lib/pulp" after upgrading Red Hat Satellite 6

Environment

• Red Hat Satellite 6.2
• Red Hat Satellite 6.3

Issue

• Pulp broken symlinks are generated in /var/lib/pulp/ after upgrading Satellite server from version 6.2.9 or older to 6.2.10/6.3 or newer
• This affects both yum clients getting 404 error on repository metadata (usually productid.gz) and also it causes capsule sync failures due to the same cause beneath

Resolution

• Prepare a rake script with commands to trigger metadata regenerate tasks for affected repositories:

echo "User.current = User.anonymous_admin" > republish_broken_repos.rb

for repo in $(find /var/lib/pulp/published/yum/master/ -type l -exec sh -c "file -b {} | grep -q ^broken" \; -print | cut -d/ -f9 | sort -u); do
  echo "ForemanTasks.async_task(Actions::Katello::Repository::MetadataGenerate, ::Katello::Repository.where(:pulp_id => \"$repo\").first,  :force => true)"
done >> republish_broken_repos.rb

• Execute it under rake console:

cat republish_broken_repos.rb | foreman-rake console

• Wait until tasks with label Actions::Katello::Repository::MetadataGenerate complete

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

• The issue occurred because some repositories failed to publish during Red Hat Satellite upgrade.

Diagnostic Steps

• On client host yum update throws below error message.

https://satellite.example.com/pulp/repos/redhat/Production/rhel_7_2_x86_64_base/content/dist/rhel/server/7/7.2/x86_64/os/repodata/productid: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
https://satellite.example.com/pulp/repos/redhat/Production/rhel_7_2_x86_64_base/content/dist/rhel/server/7/7.2/x86_64/supplementary/os/repodata/productid: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
https://satellite.example.com/pulp/repos/redhat/Production/rhel_7_2_x86_64_base/content/dist/rhel/server/7/7.2/x86_64/oracle-java/os/repodata/productid: [Errno 14] HTTPS Error 404 - Not Found
Trying other mirror.
https://satellite.example.com/pulp/repos/redhat/Production/rhel_7_2_x86_64_base/content/dist/rhel/server/7/7.2/x86_64/rhscl/1/os/repodata/productid: [Errno 14] HTTPS Error 404 - Not Found

- optionally, Capsule sync fails with:

- in both cases, `/var/log/messages` contains:

Aug 09 08:50:47 satellite.example.com pulp_streamer[84699]: [-] 127.0.0.1 - - [09/Aug/2018:07:50:47 +0000] "GET /var/lib/pulp/content/units/yum_repo_metadata_file/41/965e9dac40ed665f502493f0a7cd2f6cba184ff7f92dffba0ef63f8b3ac3e6/productid HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 Linux/3.10.0-862.el7.x86_64"

• Execute below command in Red Hat Satellite server to get list of broken symlinks in Pulp to a metadata file.

# find  /var/lib/pulp/published/yum/master/ -type l -exec sh -c "file -b {} | grep -q ^broken | grep -v rpm$" \; -print

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.