[Sat6] Capsule is terminated during synchronization of container images

Solution Unverified - Updated

Environment

  • Satellite 6
  • Satellite-Capsule 6
  • Container repositories enabled

Issue

  • Unable to fetch a container image when attached to a Satellite-capsule server.
  • Capsule-sync is terminated by a SIGKILL
  • Missing container images when using a Satellite-capsule server.

Resolution

Unless the re-sync functionality requested by This content is not included.Bz#1548966 is provided, please open a support ticket in order to get advises how to re-download the missing files. Broken or empty manifest-files may cause trouble on later upgrades.

To identify missing images on the Satellite server, please follow the steps outlined below:

# cd /var/lib/pulp/published/docker/
# find . -xtype l

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues

Root Cause

Due to the missing functionality of a re-sync for container repositories, Satellite may have broken symbolic-links in /var/www/pub/ which are then causing the web-service to return a 403 to the Satellite-caspule.

Diagnostic Steps

If there are broken links to container images on the Satellite, the following messages may be shown in the /var/log/messages file on a capsule-server:

Oct 15 14:33:55 capsule.example.com pulp[4218]: nectar.downloaders.threaded:INFO: Download failed: Download of https://satellite.example.com:5000/v2/example-ocp10-ocpcontainer-rhcc-openshift3_ose-metrics-cassandra/blobs/sha256:91e5bcf5dd294bcd9fc86b5b27e4a820d3399194dd0bee057277a4ad8b4c44bf failed with code 403: Forbidden
Oct 15 14:33:55 capsule.example.com pulp[1974]: celery.worker.job:ERROR: (1974-27456) Task pulp.server.managers.repo.sync.sync[d80d0fe0-2dad-4bf2-bb6c-b167ce5ebf39] raised unexpected: WorkerLostError('Worker exited prematurely: signal 9 (SIGKILL).',)
Oct 15 14:33:55 capsule.example.com pulp[1974]: celery.worker.job:ERROR: (1974-27456) Traceback (most recent call last):
Oct 15 14:33:55 capsule.example.com celery[1974]: reserved_resource_worker-1@capsule.example.com ready.
Oct 15 14:33:55 capsule.example.com pulp[1974]: celery.worker.job:ERROR: (1974-27456)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1171, in mark_as_worker_lost
Oct 15 14:33:55 capsule.example.com pulp[1974]: celery.worker.job:ERROR: (1974-27456)     human_status(exitcode)),
Oct 15 14:33:55 capsule.example.com pulp[1974]: celery.worker.job:ERROR: (1974-27456) WorkerLostError: Worker exited prematurely: signal 9 (SIGKILL).
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.