Red Hat Satellite or Capsule installation\upgrade fails on pulp-manage-db step with error "pkg_resources.DistributionNotFound: kombu>=4.0.2,<5.0"

Solution Verified - Updated 14 Jun 2024

Environment

Red Hat Satellite 6.9 and below
Red Hat Satellite Capsule 6.9 and below

Issue

When trying to install a new Red Hat Satellite\Capsule server or upgrade an existing one, The satellite-installer execution fails with the following error:
```
pkg_resources.DistributionNotFound: kombu>=4.2.0,<5.0
su - apache -s /bin/bash -c pulp-manage-db failed! Check the output for error!
```

Resolution

As mentioned in the Diagnostic Steps section, Check if python2-kombu is missing or installed from a third-party repository.
```
# rpm -qi python2-kombu
```
If found missing, install it back from the correct satellite\satellite-capsule repository and then re-run the upgrade.
If found installed from a third-party vendor e.g. Fedora (EPEL) then continue with the next steps as mentioned below.
Ensure that no packages related to python\pulp\qpid\qdrouterd\kobo\amqp\kombu etc are installed from third-party vendors like Fedora (EPEL).
```
# rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n' | egrep -v "Red Hat, Inc\.|Red Hat Inc" | sort | grep Fedora
```

When found any, Take a Backup or VM Snapshot of the affected system and then refer to the example of resolution steps provided below:

   ## Unlock the packages and stop all services 
   
   # foreman-maintain packages unlock
   # foreman-maintain service stop


   ## Ensure that EPEL or any other non-redhat repositories are disabled and correct satellite\capsule specific ones are enabled.

   # rm -rf /var/cache/yum/*
   # yum repolist


   ## An example list of the most commonly found packages installed\updated from EPEL repository and would require removal. 

   # rpm -e --nodeps kobo libmspack libwebsockets python2-amqp python2-billiard python2-celery python2-jmespath python-semantic_version qpid-cpp-client qpid-cpp-server qpid-cpp-server-linearstore qpid-dispatch-router qpid-proton-c qpid-qmf qpid-tools python2-gnupg python2-httplib2 python2-qpid python2-qpid-proton python2-qpid-qmf python2-simplejson


   ## Whatever is being removed above, Its counterpart should be installed back from the correct satellite\capsule specific repositories as applicable.

   # yum install kobo libmspack libwebsockets python2-amqp python2-billiard python2-celery python2-jmespath python-semantic_version qpid-cpp-client qpid-cpp-server qpid-cpp-server-linearstore qpid-dispatch-router qpid-proton-c qpid-qmf qpid-tools python-gnupg python-httplib2 python-qpid python-qpid-proton python-qpid-qmf python-simplejson


   ## Start back the services and retry the installation or upgrade that was failing before.

   # foreman-maintain service start
   # satellite-installer --verbose

NOTE: In every scenario, the list of packages to deal with would not be the same for a Red Hat Satellite or Capsule system. And hence it's recommended to engage the This content is not included.Red Hat Technical Support for further consultation.

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

Missing python2-kombu package or installed from a third-party source.
Multiple packages related to python\pulp\qpid\qdrouterd\kobo\amqp\kombu etc are installed from a third-party source.

Diagnostic Steps

Following error was observed in /var/log/foreman-installer/{satellite,capsule}.log of the affected Red Hat Satellite\Capsule server.

2022-03-23 17:18:07 [DEBUG ] [configure] Executing with uid=apache: 'pulp-manage-db'
2022-03-23 17:18:07 [INFO  ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns: Traceback (most recent call last):
2022-03-23 17:18:07 [INFO  ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns:   File "/bin/pulp-manage-db", line 5, in <module>
2022-03-23 17:18:07 [INFO  ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns:     from pkg_resources import load_entry_point
2022-03-23 17:18:07 [INFO  ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns:   File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 3011, in <module>
2022-03-23 17:18:07 [INFO  ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns:     parse_requirements(__requires__), Environment()
2022-03-23 17:18:07 [INFO  ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns:   File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 626, in resolve
2022-03-23 17:18:07 [INFO  ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns:     raise DistributionNotFound(req)
2022-03-23 17:18:07 [INFO  ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns: pkg_resources.DistributionNotFound: kombu>=4.2.0,<5.0
2022-03-23 17:18:07 [ERROR ] [configure] 'pulp-manage-db' returned 1 instead of one of [0]
2022-03-23 17:18:07 [ERROR ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]/returns: change from 'notrun' to ['0'] failed: 'pulp-manage-db' returned 1 instead of one of [0]
2022-03-23 17:18:07 [DEBUG ] [configure] /Stage[main]/Pulp::Database/Exec[migrate_pulp_db]: Evaluated in 0.10 seconds

A similar traceback was observed in the /var/log/messages file as well around the same time.

Execute these commands on the affected Satellite\Capsule to check if python2-kombu is installed and from which repository/vendor.
```
# yum info python2-kombu
# rpm -qi python2-kombu
```

Use the following commands directly on the affected Satellite\Capsule or the sosreport of the same to check for the presence of non-Red Hat provided packages.

## On system
# rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n' | egrep -v "Red Hat, Inc\.|Red Hat Inc" | sort | grep Fedora

## On sosreport
# grep -v Inc sos_commands/rpm/package-data | egrep -v "`cat hostname`|^gpg-pubkey" | awk '{print $1,$8,$9}' | sort -nk1

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

upgrade

Category

Upgrade

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.