On Red Hat Satellite 6 client installing a package returns 404 error despite the package being in metadata

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.4 and above (until 6.9)
  • DOES NOT APPLY TO 6.10+

Issue

  • Installing or upgrading a package to a client machine fails with HTTPS Error 404 - Not Found
  • Inspecting relevant repository on Satellite 6 server's /var/lib/pulp/published:
    • Metadata contains expected information about the package that is really intended to be present in the repo
    • The symlink to /var/lib/pulp/content is broken, even for a repository with Immediate download policy
    • The /var/lib/pulp/content contains the package of given filename.
  • Is there automated way of fixing symlinks so in case there are more such instances?

Resolution

  • Note:

    • That broken symlinks don't necessarily indicate a problem, it is expected to have them in case of "On Demand" download policy of given repo and also for "Background" policy for some time.
    • To confirm a broken link is really the cause of your problems, try to fetch that particular package via https - if this fails, the broken symlink is a real issue.

  • Try this 1st :

    • Refreshing your manifest file via hammer with a manifest file or via hammer via rhn manifest cloudbased

    • Retest your node/client to Satellite 6 via 

      # subscription-manager refresh
# subscription-manager identity
# subscription-manager list --consumed
# subscription-manager config
# cat /etc/redhat-release
# subscription-manager release --list
# subscription-manager release --show
# yum list <package_name>
# yum repolist -v

  • It is not recommended to fix missing broken symlinks due to several reasons (i.e. /var/lib/pulp/content might have multiple instances of a given RPM differentiating by checksum, and linking a package from repo to the wrong package with different checksum can cause more harm).

  • Rather it is recommended to regenerate metadata of the given repository through the Satellite Web UI.

    • To regenerate metadata for a repository that is synced from the upstream repository:

      • Satellite Web UI -> Content -> Products -> Select the particular product -> Select the particular repository -> Select Action -> Republish Repository Metadata

    • To regenerate metadata for a Content View:

      • Satellite Web UI -> Content -> Content Views -> Select the particular Content View -> Click on the drop down beside Promote for the latest version -> Regenerate Repository Metadata

    • To regenerate metadata for a Composite Content View, you need to use the above step to:

      • Regenerate the metadata for all its Component Content Views first.
      • After that regenerate the metadata for the Composite Content View.

  • To regenerate metadata for a repository in the Capsule server, you can use the Pulp API.

    • Run the command below on the Satellite. In below commands, use proper repository relative path from the yum output:

      # su - postgres -c "psql foreman -c \"select pulp_id from katello_repositories where relative_path = 'Red_Hat/Library/content/dist/rhel/server/7/7Server/x86_64/sat-capsule/6.2/os';\""

    • Once having the pulp repository name, run the following command on the Capsule:

      # pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)
# repo=566ff12f-cd80-49b5-a63e-1d8ab85cdbe3   #### customize this per pulp_id value
# curl -i -H "Content-Type: application/json" -X POST -d "{\"id\":\"$repo\",\"override_config\":{\"force_full\":true}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/$repo/actions/publish/

      After that, it will trigger a task in pulp. Make sure it completes successful by checking its state:

       #### replace the task id from previous API call response
 curl -i -H "Content-Type: application/json" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/tasks/<task id>/

For more KB articles/solutions related to Red Hat Satellite 6.x Repository Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Repository Issues.

Root Cause

Diagnostic Steps

  • Installing, reinstalling or upgrading a package fails with:

Content from satellite.example.com is not included.https://satellite.example.com/pulp/repos/Red_Hat/Library/content/dist/rhel/server/7/7Server/x86_64/sat-capsule/6.2/os/gofer-2.7.6-1.el7sat.noarch.rpm:%20[Errno%2014]%20HTTPS%20Error%20404%20-%20Not%20Found`

  • Checking content of /var/lib/pulp/published/yum/https/repos/Red_Hat/Library/content/dist/rhel/server/7/7Server/x86_64/sat-capsule/6.2/os directory on Satellite (cf. failed URL):

    • Metadata contains proper info about the package.
    • The filename is a broken symlink:
    # file gofer-2.7.6-1.el7sat.noarch.rpm 
gofer-2.7.6-1.el7sat.noarch.rpm: broken symbolic link to `/var/lib/pulp/content/units/rpm/01/d91ab1958314d44ce81038015cb83909aa7d968eb26e82c2791da9e21e5d7c/gofer-2.7.6-1.el7sat.noarch.rpm

    • The filename exists somewhere else under /var/lib/pulp/content/:

       # find /var/lib/pulp/content -name gofer-2.7.6-1.el7sat.noarch.rpm
/var/lib/pulp/content/units/rpm/02/d91ab1958314d44ce81038015cb83909aa7d968eb26e82c2791da9e21e5d7c/gofer-2.7.6-1.el7sat.noarch.rpm
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.