On Red Hat Satellite 6, /var/lib/pulp/content consumes much more space than expected

Environment

• Red Hat Satellite 6

Issue

• /var/lib/pulp/content consumes much more space than we would expect
• We see some RPMs stored multiple times there
• Some RPMs or ISOs found there are not known to be belonging to a synchronized repository
• Is there a way to identify what files might be redundant and what might be even deleted?

Resolution

No file under /var/lib/pulp shall be manually deleted/moved/modified without explicit confirmation from Red Hat support or pulp engineering. Unwanted removal might cause harm and will almost surely hide evidence of what potential error caused some orphaned files there.

It is recommended to:

• Clean orphaned content (weekly cron job does so, but worth running it before next step either way):

/etc/cron.weekly/pulp-maintenance

(some older Sat versions have /etc/cron.weekly/katello-remove-orphans instead, both scripts do the same thing)

Wait approx. 5 minutes to see log like Task pulp.server.managers.content.orphan.delete_all_orphans[3293ca05-2e15-4ac2-a159-e61b6e4640b0] succeeded in 123.425801985897s: None in /var/log/messages to confirm the task completed.

• Run the attached script that will (for each type of unit like RPM or ISO or distribution):

  • Identify the filenames of the units that mongo is aware of
  • Find files under /var/lib/pulp/content
  • Identify what files under that directory are not known to mongo (hence they are orphaned)

• The script shall ideally be run with pulp and httpd services stopped, otherwise it can provide false alarms (a unit downloaded in the meantime e.g. due to deferred download).

• Create an auxiliary working directory, copy the script there and run it from that directory:

# chmod +x find_orphans_in_var_lib_pulp_content.sh
# mkdir /tmp/find_orphans_in_var_lib_pulp_content
# cp find_orphans_in_var_lib_pulp_content.sh /tmp/find_orphans_in_var_lib_pulp_content
# cd /tmp/find_orphans_in_var_lib_pulp_content
# ./find_orphans_in_var_lib_pulp_content.sh
# cd ..
# tar cvzf find_orphans_in_var_lib_pulp_content.tar.gz find_orphans_in_var_lib_pulp_content
# rm -rf find_orphans_in_var_lib_pulp_content

• Provide the find_orphans_in_var_lib_pulp_content.tar.gz tarball to Red Hat support

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

• Some RPMs can be stored multiple times, especially if they differ by checksum type. Some units (RPMs, SRPMs, ISOs,..) can be present as orphans (repo recently deleted, units not yet) - weekly cronjob /etc/cron.weekly/katello-remove-orphans shall remove those orphans.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.