Unable to sync repositories on capsule from satellite 6 server

Environment

• Red Hat Satellite 6.0.8

Issue

• On capsule server connected to Satellite server, getting lot of puppet error messages in /var/log/messages.
• Unable to sync repositories on capsule from satellite 6 server. In this scenario mounting /var/lib/pulp as an NFS share, could that be an issue?
• Error in /var/log/messages :

Jul 24 02:09:12 xyz-sat pulp: pulp.server.webservices.middleware.exception:ERROR: MissingResource: Missing resource(s): distributor=Unix_Administration-Red_Hat_Software_Collections__for_RHEL_Server_-Red_Hat_Software_Collections_R
PMs_for_Red_Hat_Enterprise_Linux_6_Server_x86_64_6Server_clone
Jul 24 02:09:12 xyz-sat pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.importer.update_importer_config[5cd72d66-f68a-429b-859c-2b2b5d20d9d7] succeeded in 0.0425202739425s: {u'repo_id':...
Jul 24 02:09:12 xyz-sat pulp: pulp.server.webservices.middleware.exception:ERROR: [Errno 13] Permission denied: '/var/lib/pulp/working'
Jul 24 02:09:12 xyz-sat pulp: pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent call last):

Jul 24 02:33:21 xyz-sat puppet-agent[4708]: (/File[/var/lib/puppet/lib]) Could not evaluate: Could not retrieve information from environment production source(s) puppet://xyz-sat.us.abc.com/plugins
Jul 24 02:33:24 xyz-sat puppet-master[13355]: Compiled catalog for xyz-sat.us.abc.com in environment production in 0.02 seconds
Jul 24 02:33:24 xyz-sat puppet-agent[4708]: Finished catalog run in 0.13 seconds

Resolution

• This issue seems mostly occurs due to following reasons :

  1. If the incorrect ownership, permissions are set on the directory.
  2. The issue can also occur if the there is no enough free space on the disk.
  3. Incorrect selinux policies set on the directories.

• Verify the permissions and selinux policies set on the directory :

# audit2allow -a

#============= httpd_t ==============

#!!!! This avc can be allowed using the boolean 'httpd_use_nfs'
allow httpd_t nfs_t:dir write;

• Resolved through :

# setsebool -P httpd_use_nfs 1

• After that from satellite server tried the sync the repository and content is populating on capsule server:

# hammer capsule content synchronize --id 2 --environment-id 2

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues

Root Cause

• Selinux permissions on the directory was not set.

Diagnostic Steps

• Try disabling SELinux.

• In the log file:

# less /var/log/messages :

Jul 21 08:23:49 xyz-sat pulp: celery.worker.consumer:ERROR: consumer: Cannot connect to qpid://guest@xyz-sat.us.abc.com:5671//: [Errno 1] _ssl.c:492: error:14094418:SSL routines:SSL3_READ_BYTES:tlsv1 alert unknown c
a.
Jul 21 08:23:49 xyz-sat pulp: celery.worker.consumer:ERROR: Trying again in 32.00 seconds...
Jul 21 08:23:49 xyz-sat pulp: celery.worker.consumer:ERROR: 
Jul 21 08:23:53 xyz-sat qpidd[8299]: 2015-07-21 08:23:53 [System] error Error reading socket: Success(0)


Jul 24 02:09:12 xyz-sat pulp: pulp.server.webservices.middleware.exception:ERROR: MissingResource: Missing resource(s): distributor=Unix_Administration-Red_Hat_Software_Collections__for_RHEL_Server_-Red_Hat_Software_Collections_R
PMs_for_Red_Hat_Enterprise_Linux_6_Server_x86_64_6Server_clone
Jul 24 02:09:12 xyz-sat pulp: celery.worker.job:INFO: Task pulp.server.managers.repo.importer.update_importer_config[5cd72d66-f68a-429b-859c-2b2b5d20d9d7] succeeded in 0.0425202739425s: {u'repo_id':...
Jul 24 02:09:12 xyz-sat pulp: pulp.server.webservices.middleware.exception:ERROR: [Errno 13] Permission denied: '/var/lib/pulp/working'
Jul 24 02:09:12 xyz-sat pulp: pulp.server.webservices.middleware.exception:ERROR: Traceback (most recent call last):



Jul 24 02:33:21 xyz-sat puppet-agent[4708]: (/File[/var/lib/puppet/lib]) Could not evaluate: Could not retrieve information from environment production source(s) puppet://xyz-sat.us.abc.com/plugins
Jul 24 02:33:24 xyz-sat puppet-master[13355]: Compiled catalog for xyz-sat.us.abc.com in environment production in 0.02 seconds
Jul 24 02:33:24 xyz-sat puppet-agent[4708]: Finished catalog run in 0.13 seconds

Jul 28 02:15:21 xyz-sat pulp: pulp.server.webservices.middleware.exception:ERROR: OSError: [Errno 13] Permission denied: '/var/lib/pulp/working/repos/Unix_Administration-Red_Hat_Enterprise_Linux_Workstation-Red_Hat_Enterprise_Linux_7_Workstation_RPMs_x86_64_7Workstation/distributors'
Jul 28 02:15:21 xyz-sat goferd: [ERROR][worker-0] pulp_node.handlers.strategies:175 - Unix_Administration-Red_Hat_Enterprise_Linux_Workstation-Red_Hat_Enterprise_Linux_7_Workstation_RPMs_x86_64_7Workstation
Jul 28 02:15:21 xyz-sat goferd: [ERROR][worker-0] pulp_node.handlers.strategies:175 - Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/pulp_node/handlers/strategies.py", line 166, in _merge_repositories child.merge(parent) File "/usr/lib/python2.6/site-packages/pulp_node/handlers/model.py", line 223, in merge self.merge_distributors(repository) File "/usr/lib/python2.6/site-packages/pulp_node/handlers/model.py", line 278, in merge_distributors distributor_in.add() File "/usr/lib/python2.6/site-packages/pulp_node/handlers/model.py", line 383, in add self.dist_id) File "/usr/lib/python2.6/site-packages/pulp/bindings/repository.py", line 252, in create return self.server.POST(path, data) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 99, in POST return self._request('POST', path, body=body, ensure_encoding=ensure_encoding) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 151, in _request self._handle_exceptions(response_code, response_body) File "/usr/lib/python2.6/site-packages/pulp/bindings/server.py", line 189, in _handle_exceptions raise exceptions.PulpServerException(response_body) PulpServerException: RequestException: POST request on /pulp/api/v2/repositories/Unix_Administration-Red_Hat_Enterprise_Linux_Workstation-Red_Hat_Enterprise_Linux_7_Workstation_RPMs_x86_64_7Workstation/distributors/ failed with 500 - Unhandled Exception
Jul 28 02:15:21 xyz-sat goferd: [ERROR][worker-0] pulp_node.handlers.handler:175 - An unexpected error occurred.  repo_id=Unix_Administration-Red_Hat_Enterprise_Linux_Workstation-Red_Hat_Enterprise_Linux_7_Workstation_RPMs_x86_64_7Workstation

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.