Red Hat Capsule synchronization fails with error RPM1004: Error retrieving metadata: 'X509' object has no attribute '_x509'

Solution Verified - Updated

Environment

  • Red Hat Capsule 6

Issue

  • Attempt to sync contents on a capsule server fails with following error.

    Exception:
Katello::Errors::PulpError: RPM1004: Error retrieving metadata: 'X509' object has no attribute '_x509'
Backtrace:
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.16/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.16/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.0.16/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'

Resolution

Before proceeding with the resolution

CAUTION: It's recommended to take a Backup or VMware snapshot of the affected Satellite\Capsule server before proceeding any further with the cleanup process.

  • Go through the Diagnostic Steps section to identify the third-party module causing the problem and it's most likely the requests module.

  • Execute following steps on affected capsule server to install back the correct requests module from python-requests package.

    # foreman-maintain service stop
# pip uninstall requests         ## If needed please use full path of pip binary here
# yum reinstall python-requests -y --disableplugin=foreman-protector
# rpm -V python-requests python-urllib3

  • Following file and directory should not be present at this moment on the capsule server.

    # ls -ld /usr/lib/python2.7/site-packages/requests-2.12.1-py2.7.egg/requests/packages/urllib3/contrib/pyopenssl.py  /usr/lib/python2.7/site-packages/requests-2.12.1-py2.7.egg

    • If true, then start back capsule services and verify their health

      # foreman-maintain service start
# foreman-maintain service status

  • Initiate an Optimized sync for the affected Capsule server and monitor the progress.

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues

Root Cause

  • The python2-pip package was installed from the EPEL repository and the requests module was upgraded to a higher version using pip binary.

  • In some scenarios the file could be a bit different than what was mentioned in the Diagnostic Steps section and could be related to the urllib3 module as well.

Diagnostic Steps

  • After the capsule sync task has failed, look for similar messages as below in the /var/log/messages file of the affected capsule server.

    Aug 27 16:39:51 capsule pulp: nectar.downloaders.threaded:ERROR: (2431-87168)     cert = self.sock.getpeercert()
Aug 27 16:39:51 capsule pulp: nectar.downloaders.threaded:ERROR: (2431-87168)   File "/usr/lib/python2.7/site-packages/requests-2.12.1-py2.7.egg/requests/packages/urllib3/contrib/pyopenssl.py", line 324, in getpeercert
Aug 27 16:39:51 capsule pulp: nectar.downloaders.threaded:ERROR: (2431-87168)   File "/usr/lib/python2.7/site-packages/requests-2.12.1-py2.7.egg/requests/packages/urllib3/contrib/pyopenssl.py", line 324, in getpeercert
Aug 27 16:39:51 capsule pulp: nectar.downloaders.threaded:ERROR: (2431-87168)     'subjectAltName': get_subj_alt_name(x509)
Aug 27 16:39:51 capsule pulp: nectar.downloaders.threaded:ERROR: (2431-87168)     'subjectAltName': get_subj_alt_name(x509)
Aug 27 16:39:51 capsule pulp: nectar.downloaders.threaded:ERROR: (2431-87168)   File "/usr/lib/python2.7/site-packages/requests-2.12.1-py2.7.egg/requests/packages/urllib3/contrib/pyopenssl.py", line 166, in get_subj_alt_name
Aug 27 16:39:51 capsule pulp: nectar.downloaders.threaded:ERROR: (2431-87168)   File "/usr/lib/python2.7/site-packages/requests-2.12.1-py2.7.egg/requests/packages/urllib3/contrib/pyopenssl.py", line 166, in get_subj_alt_name
Aug 27 16:39:51 capsule pulp: nectar.downloaders.threaded:ERROR: (2431-87168)     cert = _Certificate(openssl_backend, peer_cert._x509)
Aug 27 16:39:51 capsule pulp: nectar.downloaders.threaded:ERROR: (2431-87168)     cert = _Certificate(openssl_backend, peer_cert._x509)
Aug 27 16:39:51 capsule pulp: nectar.downloaders.threaded:ERROR: (2431-87168) AttributeError: 'X509' object has no attribute '_x509'
Aug 27 16:39:51 capsule pulp: nectar.downloaders.threaded:ERROR: (2431-87168) AttributeError: 'X509' object has no attribute '_x509'

  • As it can be seen the file from where the error is being raised is the pyopenssl.py file. Identify if that file is provided by any of the installer packages on the capsule.

    # rpm -qf /usr/lib/python2.7/site-packages/requests-2.12.1-py2.7.egg/requests/packages/urllib3/contrib/pyopenssl.py

  • In most scenarios this file will not be associated with any package or even if it is, that package will be a non-redhat package.

  • In the situation where the file is not associated with any packages, identify if pip or any other third-party packages are installed on the affected capsule.

       --> From Command line:

   # yum list installed python2-pip python3-pip
   # rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH} %{VENDOR}\n' | grep -v 'Red Hat, Inc\.' | sort


   --> From sosreport of the capsule.

   $ grep -v Inc sos_commands/rpm/package-data | grep -v `cat hostname` | awk '{print $1,$8, $9}' | grep Fedora
   python2-pip-8.1.2-12.el7.noarch Fedora Project
   python2-qpid-proton-0.31.0-3.el7.x86_64 Fedora Project
   iperf-2.0.13-1.el7.x86_64 Fedora Project
   libmspack-0.7-0.1.alpha.el7.x86_64 Fedora Project
   nmon-16g-3.el7.x86_64 Fedora Project
   lyx-fonts-2.2.3-1.el7.noarch Fedora Project
   qpid-proton-c-0.31.0-3.el7.x86_64 Fedora Project

  • Notice that the installed version of the python-requests package differs from the installed requests module reported by pip.

       --> From command line:

   # rpm -q python-requests python-urllib3
   python-requests-2.6.0-10.el7.noarch
   python-urllib3-1.10.2-7.el7.noarch

   # pip list | egrep "request|urllib3"
   requests (2.12.1)
   urllib3 (1.10.2)

   --> From sosreport of capsule

   # egrep "request|urllib3" installed-rpms 
   python-requests-2.6.0-10.el7.noarch
   python-urllib3-1.10.2-7.el7.noarch

   # egrep "request|urllib3" sos_commands/python/pip_list
   requests (2.12.1)
   urllib3 (1.10.2)
    • As it can be noticed, the version of urllib3 module package and module matches but pip shows a higher version of the requests module installed.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.