Capsule synchronization fails on a docker content every time

Solution Verified - Updated

Environment

Red Hat Satellite 6

Issue

  • synchronizing a Capsule every times fails
  • a docker repository is always failed to be synchronized
  • Capsule logs show WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL) error every time

Resolution

Identify what docker blobs (or potentially other content) on Satellite has insufficient permissions, and fix that.

Check in Capsule for log in /var/log/messages:

Dec 14 16:17:34 caps63 pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://sat63.example.com:5000/v2/redhat-docker-centos2/blobs/sha256:004fc2c6daa54a848a5c7af9e16a7a9f5edfd984315b0d270a43e794987ef6db failed with code 403: Forbidden

Go to Satellite, and ensure proper permissions, symlink ownership and SELinux context is established for that file (2nd line contains suffix of above URI after /v2/):

# cd /var/lib/pulp/published/docker/v2/web
# ls -lZ redhat-docker-centos2/blobs/sha256:004fc2c6daa54a848a5c7af9e16a7a9f5edfd984315b0d270a43e794987ef6db
lrwxrwxrwx. apache apache system_u:object_r:httpd_sys_rw_content_t:s0 redhat-docker-centos2/blobs/sha256:004fc2c6daa54a848a5c7af9e16a7a9f5edfd984315b0d270a43e794987ef6db -> /var/lib/pulp/content/units/docker_blob/e1/f56c82d6e990d5abc556e6acfe6dd7af42cbdb3c2b4a9c39c31eba4d3ad2b3/sha256:004fc2c6daa54a848a5c7af9e16a7a9f5edfd984315b0d270a43e794987ef6db
#

Then, ensure the target file permissions/ownership/context:

# ls -lZ /var/lib/pulp/content/units/docker_blob/e1/f56c82d6e990d5abc556e6acfe6dd7af42cbdb3c2b4a9c39c31eba4d3ad2b3/sha256:004fc2c6daa54a848a5c7af9e16a7a9f5edfd984315b0d270a43e794987ef6db
-rw-r--r--. apache apache system_u:object_r:httpd_sys_rw_content_t:s0 /var/lib/pulp/content/units/docker_blob/e1/f56c82d6e990d5abc556e6acfe6dd7af42cbdb3c2b4a9c39c31eba4d3ad2b3/sha256:004fc2c6daa54a848a5c7af9e16a7a9f5edfd984315b0d270a43e794987ef6db

Fix if either permission prevents accessing the symlink or the file.

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues

Diagnostic Steps

/var/log/messages on Capsule having:

Dec 14 16:17:34 caps63 pulp: nectar.downloaders.threaded:INFO: Download failed: Download of https://sat63.example.com:5000/v2/redhat-docker-centos2/blobs/sha256:004fc2c6daa54a848a5c7af9e16a7a9f5edfd984315b0d270a43e794987ef6db failed with code 403: Forbidden
..
Dec 14 16:17:34 caps63 pulp: celery.worker.job:ERROR: (10505-18752) Task pulp.server.managers.repo.sync.sync[5249c29b-b328-48c5-b97a-84e65cc815d3] raised unexpected: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).',)
Dec 14 16:17:34 caps63 pulp: celery.worker.job:ERROR: (10505-18752) Traceback (most recent call last):
Dec 14 16:17:34 caps63 celery: reserved_resource_worker-1@caps63.example.com ready.
Dec 14 16:17:34 caps63 pulp: celery.worker.job:ERROR: (10505-18752)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1171, in mark_as_worker_lost
Dec 14 16:17:34 caps63 pulp: celery.worker.job:ERROR: (10505-18752)     human_status(exitcode)),
Dec 14 16:17:34 caps63 pulp: celery.worker.job:ERROR: (10505-18752) WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).
..
Dec 14 16:17:34 pmoravec-caps63 pulp: pulp.server.async.tasks:INFO: [6c78e828] Task failed : [5249c29b-b328-48c5-b97a-84e65cc815d3] : Worker terminated abnormally while processing task 5249c29b-b328-48c5-b97a-84e65cc815d3.  Check the logs for details
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.