The upgrade to Red Hat Satellite 6.9 fails with error 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.9

Issue

  • Upgrade to Red Hat Satellite 6.9 fails with the following error:

    2022-01-04 16:38:19 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-01-04 16:38:19 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]

Resolution

  • Ensure that no python-specific binaries\modules\libraries\directory_structures are created inside /usr/local/lib/ directory of Red Hat Satellite server.

    • By default, the directory /usr/local/lib/ is expected to be blank in any Red Hat Satellite or Capsule installations.

    • If there are any contents found inside /usr/local/lib/ , they should be moved to a different place or probably renamed temporarily before the upgrade of the satellite can be re-attempted.

  • Ensure that pip or pip3 binary was not used to install\upgrade\downgrade any python modules manually on Red Hat Satellite server or else it would cause requirement conflicts for concerned python modules during upgrade process of Red Hat Satellite server.

  • Additionally if an error related to importing six python module is reported in the logs, ensure that the following files are not present in the affected Satellite server.

    # ls -l /usr/bin/six*
lrwxrwxrwx. 1 root root 22 Mar 17  2017 /usr/bin/six.pyc -> /usr/local/bin/six.pyc

 

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

  • There was a different set of python binaries\modules\libraries\directory_structure present inside /usr/local/lib/ directory with updated python modules. By nature , When updated modules are available, Python will always look into this path and try to use those updated modules but the components of satellite i.e. Pulp, may not be compatible with the same.

Diagnostic Steps

  • During the upgrade failure, the following types of errors were logged inside /var/log/foreman-installer/satellite.log on two different occurrences.

    2021-12-16 15:13:48 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 779, in resolve
2021-12-16 15:13:48 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     raise VersionConflict(dist, req).with_context(dependent_req)
2021-12-16 15:13:48 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: pkg_resources.ContextualVersionConflict: (idna 3.3 (/usr/local/lib/python3.6/site-packages), Requirement.parse('idna<3,>=2.5'), {'requests'})

    or

    2022-01-04 16:38:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     for dist in factory(fullpath):
2022-01-04 16:38:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2020, in distributions_from_metadata
2022-01-04 16:38:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     if len(os.listdir(path)) == 0:
2022-01-04 16:38:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/site-packages/urllib3-1.26.4.dist-info'
    • As it can be observed from the last line of both of the errors, The issue is either with version conflict of python module or inaccessible path of the required module but the common fact in both of them is the location of the module i.e. /usr/local/lib/python3.6/site-packages which is not the expected place for python modules to be present on Satellite server.

 

  • Another type of error that can be observed is as below i.e.

       2022-01-12 12:03:48 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     import dateutil.parser
   2022-01-12 12:03:48 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-
packages/dateutil/parser/__init__.py", line 2, in <module>
   2022-01-12 12:03:48 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     from ._parser import parse, 
parser, parserinfo, ParserError
   2022-01-12 12:03:48 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-
packages/dateutil/parser/_parser.py", line 42, in <module>
   2022-01-12 12:03:48 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     import six
   2022-01-12 12:03:48 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: ImportError: bad magic number in
 'six': b'\x03\xf3\r\n'
   2022-01-12 12:03:48 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
    • This error is very specific to the six pyhon module and the presence of some bad pyc files related to the same.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.