How to verify integrity of /var/lib/pulp on Satellite or Capsule since 6.10?

Environment

Red Hat Satellite or Capsule 6.10+

Issue

We had or suspect some data corruption of /var/lib/pulp. How to verify all artifacts there are in a good shape?

Resolution

One option is to run Verify Checksum for each and every repository (on Satellite only). Optionally, one can use below script verify_artifacts_on_disk.py:

from pulpcore.app.models import Artifact
from os import environ, path
import hashlib

TRUE_VALUES = ('1', 'True', 'true', 'yes', 'Yes')
ROOTDIR = environ.get('ROOTDIR', '/var/lib/pulp/media')
CHECKSIZE = environ.get('CHECKSIZE', False) in TRUE_VALUES
CHECKSHA256 = environ.get('CHECKSHA256', False) in TRUE_VALUES

def get_sha256_for_file(fpath):
    sha256 = hashlib.sha256()
    with open(fpath, 'rb') as f:
        sha256.update(f.read())
    return sha256.hexdigest()

print(f"Processing {Artifact.objects.count()} artifacts..")
for a in Artifact.objects.all():
    fpath = path.join(ROOTDIR, a.file.path)
    if not path.isfile(fpath):
        print(f"Artifact with pulp_id {a.pulp_id} (size {a.size}, sha256 {a.sha256}) is missing at {fpath}!")
        continue
    if CHECKSIZE and path.getsize(fpath) != a.size:
        print(f"Artifact with pulp_id {a.pulp_id} at {fpath} has different size ({path.getsize(fpath)}) than expected ({a.size})!")
    if CHECKSHA256 and a.sha256:
        fpath_sha256 = get_sha256_for_file(fpath)
        if fpath_sha256 != a.sha256:
            print(f"Artifact with pulp_id {a.pulp_id} at {fpath} has different sha256 than expected ({a.sha256})!")

print("DONE")

Run it as follows:

# cat verify_artifacts_on_disk.py | sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell
Processing 23632 artifacts..
DONE
#

By default, the script checks for presence of all files only. To check also their size, add CHECKSIZE=true argument like:

cat verify_artifacts_on_disk.py | sudo -u pulp CHECKSIZE=true PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell

And to verify also their sha256 sum, use option CHECKSHA256=true the same way. Be aware this is a lengthy and CPU demanding option, as the checksum will be calculated for all files under /var/lib/pulp/media!

For more KB articles/solutions related to usage of pulpcore-manager script, please refer to How to use pulpcore-manager since Satellite 6.10.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 3.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 3.0-related Issues.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.