How to restore missing or corrupted docker_blob file in Satellite6?

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.9

Issue

  • Satellite 6 synchronized a docker repository but some downloaded file is missing, like:

    # file /var/lib/pulp/published/docker/v2/web/fc5c5240-a3cf-4def-947a-768d0a86f0e3/blobs/sha256:232aa1c3998b39ad79846afe4415af229962c62ca70c793d3b344e91336ca8bd
/var/lib/pulp/published/docker/v2/web/fc5c5240-a3cf-4def-947a-768d0a86f0e3/blobs/sha256:232aa1c3998b39ad79846afe4415af229962c62ca70c793d3b344e91336ca8bd: broken symbolic link to `/var/lib/pulp/content/units/docker_blob/3a/957415e7fc67eb36b3ad2dfcf403824d06f9761a474da21ed61e41a7b9410e/sha256:232aa1c3998b39ad79846afe4415af229962c62ca70c793d3b344e91336ca8bd'

  • Or the downloaded file is corrupted (checksum does not match the filename):

    # sha256sum /var/lib/pulp/content/units/docker_blob/3a/957415e7fc67eb36b3ad2dfcf403824d06f9761a474da21ed61e41a7b9410e/sha256:232aa1c3998b39ad79846afe4415af229962c62ca70c793d3b344e91336ca8bd
28d27596e201fcd5bcb41e62e5d7ac6cb182acb746d70ce57d5f49cf2e218d8b  /var/lib/pulp/content/units/docker_blob/3a/957415e7fc67eb36b3ad2dfcf403824d06f9761a474da21ed61e41a7b9410e/sha256:232aa1c3998b39ad79846afe4415af229962c62ca70c793d3b344e91336ca8bd

  • How to re-download the file, if forceful sync of the repository does not work?

Resolution

Since the below procedure updates Mongo database directly, it is highly recommended to create a satellite backup or VM snapshot before continue.

  • If some file is missing, identify its destination by detecting broken symlinks:

    # find /var/lib/pulp/published/docker/v2/web/ -xtype l

  • from the above output, identify the "base" repository (within web directory, it will have the shortest name in above output, like fc5c5240-a3cf-4def-947a-768d0a86f0e3 in our case) - use that name below

Initial steps for malformed file content:

  • if "just" some downloaded file has incorrect checksum, recall its destination. To identify pulp repository ID, navigate in WebUI to the problematic repository and see "Backend Identifier"

Further steps to fix both scenarios

  • once you identified the file path and repository ID, connect to mongo shell and remove the missing unit as well as with its association to the repository (replace the REPO_ID by yours)

    # mongo pulp_database

> db.units_docker_blob.find({'_storage_path':'/var/lib/pulp/content/units/docker_blob/3a/957415e7fc67eb36b3ad2dfcf403824d06f9761a474da21ed61e41a7b9410e/sha256:232aa1c3998b39ad79846afe4415af229962c62ca70c793d3b344e91336ca8bd'})

    (this will return one record with { "_id" : "084807a1-4125-413f-97bf-aa4505cd7c01", .. - replace UNIT_ID by yours _id in below command)

    > db.repo_content_units.find({'repo_id': 'REPO_ID', 'unit_id': 'UNIT_ID'})

    (the above should return just one record, if so, let delete it)

    > db.repo_content_units.remove({'repo_id': 'REPO_ID', 'unit_id': 'UNIT_ID'})

> db.units_docker_blob.remove({'_storage_path':'/var/lib/pulp/content/units/docker_blob/3a/957415e7fc67eb36b3ad2dfcf403824d06f9761a474da21ed61e41a7b9410e/sha256:232aa1c3998b39ad79846afe4415af229962c62ca70c793d3b344e91336ca8bd'})

  • now, synchronize the repository again:

    > repo=fc5c5240-a3cf-4def-947a-768d0a86f0e3  # use your own REPO_ID
pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

> curl -i -H "Content-Type: application/json" -X POST -d "{\"override_config\":{\"force_full\":true, \"num_threads\":4, \"validate\":true}}" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/repositories/$repo/actions/sync/

    It will output something like:

    HTTP/1.1 202 Accepted
Date: Wed, 19 Dec 2018 13:03:28 GMT
Server: Apache/2.4.6 (Red Hat Enterprise Linux)
Content-Length: 172
Content-Type: application/json; charset=utf-8

{"spawned_tasks": [{"_href": "/pulp/api/v2/tasks/a4f087d3-e8eb-496b-b8a1-9b1e188e7da9/", "task_id": "a4f087d3-e8eb-496b-b8a1-9b1e188e7da9"}], "result": null, "error": null}

  • Grab that _href value and use in below command:

    > curl -i -H "Content-Type: application/json" -u admin:$pulpAdminPassword https://$(hostname -f)/pulp/api/v2/tasks/a4f087d3-e8eb-496b-b8a1-9b1e188e7da9/ | grep exception | python -m json.tool

  • Wait until the sync task is finished

  • Update other repositories (i.e. cloned from the base one to a Content View) from the old reference to deleted UNIT_ID to the new one:

    # mongo pulp_database

> db.repo_content_units.find({'unit_id': 'UNIT_ID'})

    (replace there the original unit id that you deleted; this output will list repos that still refer to the deleted unit and needs updating to the new unit)

    > db.units_docker_blob.find({'_storage_path':'/var/lib/pulp/content/units/docker_blob/3a/957415e7fc67eb36b3ad2dfcf403824d06f9761a474da21ed61e41a7b9410e/sha256:232aa1c3998b39ad79846afe4415af229962c62ca70c793d3b344e91336ca8bd'})

    (remember the _id referred as NEW_UNIT_ID further)

    > db.repo_content_units.update({'unit_id': 'UNIT_ID'}, { $set: {'unit_id': 'NEW_UNIT_ID'}}, {multi: true})

  • now all pulp repositories should be properly fixed

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.