How to verify pulp publications and distributions for katello repositories?

Solution Verified - Updated

Environment

Red Hat Satellite 6.10 or newer

Issue

During a CV publish/promote or during a repo update or sync, we are hitting errors with response code 400 and text like Response body: {"publication":["Invalid hyperlink - Object does not exist."]}. Various repositories or CVs are affected. How to detect what all of them have an issue and how to fix it?

Resolution

If you suspect just publications are missing, it is worth trying the script from this KCS which is more user-friendly approach that fixes most of these issues.

Run attached script that will - for RPM non-kickstart repos - verify if katello record has valid publication that has a distribution to the desired path:

# ./verify_repo_pub_distr.sh
repo version /pulp/api/v3/repositories/rpm/rpm/fa5be15a-8313-4edb-bfff-567bf3f9a5d7/versions/3/ with publication /pulp/api/v3/publications/rpm/rpm/35ec68b7-2b92-4716-8ed4-805fa28c4193/ lacks proper distribution to path ORG/Library/CV_CAPSULE/content/dist/rhel/server/7/7Server/x86_64/sat-capsule/6.9/os !
..
#

For each reported warning, react accordingly:

repo version with publication lacks proper distribution

Log repo version <version> with publication <publication> lacks proper distribution to path <path> ! means pulp does not know how to distribute the content to consumers.

  • If multiple repos of the same Content View (third section of the ) are affected, then try republishing the CV and optionally promoting it to the given LE
  • If a CCV is affected that way, find out what component CV is affected, fix the CV by publish/promote, and then republish/promote the CCV
  • if the above does not help (on new script run) or if the issue affects non-CV repos, then execute in foreman-rake console (substitute path* by proper paths from the script output):
smart_proxy = ::SmartProxy.pulp_primary

Katello::Repository.where(relative_path: ['path1', 'path2', 'path3',.. 'pathN']).each() do |repo|
  ma = repo.backend_service(smart_proxy, true).with_mirror_adapter
  ma.refresh_distributions
end
  • if the task triggered from refresh_distributions fail with 400 response code, then ... (nullify repo's publication_href and try again?)

repo version with rel.path .. keeps publication unknown to pulp

If script outputs repo version .. with rel.path .. keeps publication .. in katello that is unknown to pulp! then both publication and distribution are missing, so either run carefully the attached script:

sed -i 's!Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/jbeap/6.4/os!\${relpath}!1'  fix_missing_publication_distribution.sh   # there is an error in the script we cant fix in the attachment directly


./fix_missing_publication_distribution.sh /pulp/api/v3/repositories/rpm/rpm/49758ed2-6f11-48ff-9d7e-aa73aab387c7/ Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/jbeap/6.4/os

(where the first argument is pulp repository href (without version) and second argument is relative path to publish), or run the same manually (for a better control):

  • create pulp publication:
curl -H "Accept:application/json" -H "Content-Type: application/json" -X POST -d @create_pub.json -s --cacert /etc/pki/katello/certs/katello-server-ca.crt --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://$(hostname -f)/pulp/api/v3/publications/rpm/rpm/

for create_pub.json:

{
    "repository": "/pulp/api/v3/repositories/rpm/rpm/8b214449-80db-402a-88cb-3a33fedeaf76/",     # update accordingly per script output - truncate version from the repo version URI
    "metadata_checksum_type": "sha256",                                                          # skip this if katello_repositories has it blank
    "package_checksum_type": "sha256"                                                            # skip this if katello_repositories has it blank
}

output will have a task's URI; query it like:

curl -s --cacert /etc/pki/katello/certs/katello-server-ca.crt --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://$(hostname -f)/pulp/api/v3/tasks/dfb87e13-6367-46bc-b2bd-e80fc531065b/

and see there:

"created_resources":["/pulp/api/v3/publications/rpm/rpm/744c550d-09fb-4394-b2fd-2ebb974be971/"

then fire:

curl -H "Accept:application/json" -H "Content-Type: application/json" -X POST -d @create_distr.json -s --cacert /etc/pki/katello/certs/katello-server-ca.crt --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://$(hostname -f)/pulp/api/v3/distributions/rpm/rpm/

for create_distr.json:

{
    "base_path": "base/path/from/script/output",
    "content_guard": null,     # for Red Hat repos, use content guard URI like "/pulp/api/v3/contentguards/certguard/rhsm/b7c11394-6029-4656-90e3-01705d23fe89/"
    "name": "zoo_repo-78092730",
    "publication": "/pulp/api/v3/publications/rpm/rpm/744c550d-09fb-4394-b2fd-2ebb974be971/"    # update from querying task status
}

output will have a task's URI; query it like:

curl -s --cacert /etc/pki/katello/certs/katello-server-ca.crt --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key https://$(hostname -f)/pulp/api/v3/tasks/dfb87e13-6367-46bc-b2bd-e80fc531065b/

to see there:

"created_resources":["/pulp/api/v3/distributions/rpm/rpm/08be3375-5676-40ed-a2fb-ce36c1841d41/"]

Update that hrefs to the katello repository:

distr_href="/pulp/api/v3/distributions/rpm/rpm/08be3375-5676-40ed-a2fb-ce36c1841d41/"           # the created_resources distribution href
path="Default_Organization/Library/content/els/rhel/server/6/6Server/x86_64/optional/os"   # the URI path from the script
su - postgres -c "psql foreman -c \"UPDATE katello_distribution_references SET href = '${distr_href}' WHERE path = '${path}';\""

pub_href="/pulp/api/v3/publications/rpm/rpm/03effa78-cc91-4831-bcf5-f5880c48b042/"           # the created_resources publication href
su - postgres -c "psql foreman -c \"UPDATE katello_repositories SET publication_href = '${pub_href}' WHERE relative_path = '${path}';\""

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 3.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 3.0-related Issues

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.