How to verify all expected content in a Lifecycle Environment has been successfully synchronized to a Capsule?

Solution Verified - Updated

Environment

Red Hat Satellite 6 with Capsule(s)

Issue

Having a Capsule in some Lifecycle Environment (LE), A Content View promotion to the LE triggers synchronization of the new content to the Capsule. However, due to some dynflowd service restarts and some failed tasks, we are not sure if the content was properly synchronized or not.

Is there a way to verify all content on Capsule has been properly synchronized?

Resolution

First, the most easy way to verify it is to check status of the Capsule synchronization tasks. One should filter tasks to

(label = Actions::Katello::CapsuleContent::Sync or label = Actions::Katello::ContentView::CapsuleSync) and result != success

and find out possible failed or unfinished Capsule sync tasks.

Another high level way to verify whole content is properly synchronized is to invoke a Capsule Synchronization manually and let it complete. The task will skip properly synchronized repositories automatically. To limit that synchronization just to some LE, use either:

  • hammer command like:
hammer capsule content synchronize --environment-id 8 --organization-id 1 --id 2
  • or direct API like:
curl -X POST -s -k -u user:password -H "Content-Type: application/json" -d "{\"environment_id\": 8 }" "https://$(hostname -f)/katello/api/capsules/2/content/sync"

In case neither above is applicable option, or if one needs some pulp/repository content-based check, then since Sat6.10 version, create repo_content_stats.py file with content:

from pulp_rpm.app.models.repository import RpmDistribution, RpmPublication
from pulpcore.app.models.repository import RepositoryVersion
from os import environ

uri = environ['URI'] if 'URI' in environ.keys() else ''

for rpmd in RpmDistribution.objects.filter(base_path__contains=uri).all():
    print(f"{rpmd.base_path.ljust(80)} | {str(rpmd.pulp_created)} | {rpmd.publication.repository_version.content.count()}")

and run it on Satellite and Capsule via:

cat repo_content_stats.py | sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell

which will produce full URIs, timestamp of publication and count of all artifacts (RPMs+errata+..), like:

RedHat/Library/content/dist/layered/rhel8/x86_64/ansible/2.9/os                  | 2023-01-12 08:31:35.693966+00:00 | 78
RedHat/Library/content/dist/layered/rhel8/x86_64/sat-capsule/6.12/os             | 2023-01-16 11:49:03.383106+00:00 | 291
RedHat/PROD/MyCV/content/dist/rhel8/8/x86_64/appstream/os                        | 2023-02-08 13:51:35.489986+00:00 | 31825
RedHat/Library/MyCV/content/dist/rhel8/8/x86_64/appstream/os                     | 2023-02-08 13:50:32.495014+00:00 | 31825

For each URI:

  • the timestamp on Capsule should be newer than on Satellite
  • the count of artifacts should match

When interested in some Organization / Lifecycle Environment or Content View, let set URI variable accordingly, like for Lifecycle Environment name DEV:

$ cat repo_content_stats.py | sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' URI='/DEV/' pulpcore-manager shell
RedHat/PROD/MyCV/content/dist/rhel8/8/x86_64/appstream/os                        | 2023-02-08 13:51:35.489986+00:00 | 31825
$


**On Satellite/Capsule 6.9 or older:** use rather the script `le_content_audit.sh` that generates:

- list of repos in given LE (optionally also given CV only)
- timestamp of repo publish being higher on Capsule than on Satellite (otherwise Satellite might have newer content)
- package list or just package count in each repo is the same

Let customize the attached script per organization label, LE and optionally also CV label (this can be left blank), and run it on Capsule. It will generate 3 files:

LE_audit_symlinks.$(hostname -f).txt
LE_audit_timestamps.$(hostname -f).txt
LE_audit_packages.$(hostname -f).txt


Each file provides data for the three checks above.

Copy the files to Satellite, and run on Satellite:

./LE_content_audit.sh capsule.fqdn.example.com


The script will generate same files for Satellite and then run the comparison.

The script compares just yum repos published under https.

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the [Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues](https://access.redhat.com/articles/6445391)
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.