Content Host missing a package metadata - how to troubleshoot or potentially fix it?

Solution Verified - Updated

Environment

Red Hat Satellite 6.9 or older (some parts applicable to 6.10 also)

Issue

How to diagnose and potentially fix various repository metadata problem on a Content Host, registered to Satellite6 or its external Capsule? For example:

  • metadata shows zero or less than expected packages
  • package install or update fails on dependency (that should not happen in the repository)

Resolution

Be aware that some packages on RHEL8 can be "hidden" due to modularity. See this solution for more details.

The most two probable causes are explored here: yum/dnf cache having obsolete data, or pulp repository is wrongly published on Satellite or Capsule (where the Content Host is registered to).

First, try cleaning cache on the Content Host:

yum clean all
rm -rf /var/cache/yum/* /var/cache/dnf/*

Then try the same action that was failing. If it works well, you are done. If not, let check metadata on the Capsule/Satellite. In /etc/yum.repos.d/redhat.repo, find the relevant repository and its baseurl, being something like:

baseurl = https://capsule007.example.com/pulp/repos/MyOrg/DEVEL/cv_rhel/content/dist/rhel/server/7/$releasever/$basearch/os

Remember the URI sub-path after /repos/, substituting $releasever and $basearch by relevant values (i.e. 7Server and x86_64, in our example below). Go to the capsule007.example.com server where the Content Host is registered to and execute:

cd /var/lib/pulp/published/yum/https/repos/
file <the-subpath-from-baseurl>

It will be a symbolic link like:

RedHat/DEVEL/cv_rhel/content/dist/rhel/server/7/7Server/x86_64/os: symbolic link to `/var/lib/pulp/published/yum/master/yum_distributor/1-cv_rhel-DEVEL-4bf21816-6c63-4444-8e65-ad56a53201ff/1605261792.16'

The 1-cv_rhel-DEVEL-4bf21816-6c63-4444-8e65-ad56a53201ff is the pulp repository name, and 1605261792.16 is the timestamp when the repo was published the latest, in our case:

# date -d @1605261792.16
Fri Nov 13 11:03:12 CET 2020
#

The timestamp should match the time when the Capsule was synchronized the latest time. To see packages in that repo metadata:

repo=1-cv_rhel-DEVEL-4bf21816-6c63-4444-8e65-ad56a53201ff   # customize this to your repo_id
zgrep "^<location" /var/lib/pulp/published/yum/master/yum_distributor/${repo}/*/repodata/*primary*

If there are not enough packages, let forcefully re-publish the repo using direct pulp API request:

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)
curl -i -H "Content-Type: application/json" -X POST -d "{\"id\":\"$repo\",\"override_config\":{\"force_full\":true}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/$repo/actions/publish/

The above will output something like:

HTTP/1.1 202 Accepted
Date: Sat, 14 Nov 2020 13:48:16 GMT
Server: Apache
Content-Length: 172
Content-Type: application/json; charset=utf-8

{"spawned_tasks": [{"_href": "/pulp/api/v2/tasks/69339643-a4da-4522-b6af-18d1067dcc45/", "task_id": "69339643-a4da-4522-b6af-18d1067dcc45"}], "result": null, "error": null}

Wait until the task is finished:

task_id=69339643-a4da-4522-b6af-18d1067dcc45  # customize this per the previous command output
curl -ks -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/tasks/${task_id}/ | python -m json.tool | grep state

Once the output contains "state": "finished",, the metadata should be regenerated, contain relevant packages and yum/dnf client should get them properly.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.