Satellite6 fills /var/lib/qpidd directory when registering and unregistering content hosts

Environment

Red Hat Satellite 6

Issue

• having Satellite6 with often content host registration and unregistration
• over the time, /var/lib/qpidd/ directory is filling up
  • there are plenty of directories named like pulp.agent.019bce30-685a-490f-b142-1263dd1a67f5 in /var/lib/qpidd/qls/jrnl (on RHEL6) or in /var/lib/qpidd/.qpidd/qls/jrnl (on RHEL7)

How to prevent the directory filling up?

Resolution

For resolution, upgrade to Satellite 6.1 GA that has fixed the underlying This content is not included.bug. In some rare cases, some queues might get orphaned even on 6.1 or newer.

As a workaround, use attached script that deletes orphaned qpid queues (see Root cause for explanation). The script:

• finds all content hosts UUIDs from foreman's database
• finds all relevant qpid queues and stores those UUIDs
• deletes queues for those no content host of given UUID exists

Script execution:

$ ./remove_orhpaned_qpid_queues.sh
finding content hosts UUIDs from foreman's db..
finding qpid queues for pulp consumers..

found 1 content hosts and 6 qpid queues for pulp consumers

list of content hosts: content-hosts-uuids.txt, list of qpid queues: qpid-queues-uuids.txt
- please remove the files once you dont need them

Please check above output for number of content hosts in each organization. If those numbers dont correspond to expected nubers, dont continue with script execution.

Delete 5 orphaned queues with no matching content host UUID (y/n)? y
deleting queue pulp.agent.4ca00772-663c-49d4-a918-31e0485f2c9c
deleting queue pulp.agent.7b6aab20-449b-41b4-a067-af3426762a0f
deleting queue pulp.agent.9fcf270e-c5b5-4c86-9dfb-66e49c9a67d0
deleting queue pulp.agent.a217e8be-f653-4bd9-80e8-f285fb7ca5ae
deleting queue pulp.agent.e73a4357-fd35-4126-bfa9-a97cc3e97a97
deleting queue pulp.agent.fbb530f1-a7de-448a-9f91-f46ddd33ccb2
$

One can review the list of content hosts and qpid queues in files content-hosts.txt and qpid-queues-uuids.txt.

Root Cause

There is a This content is not included.bug that unregistering a content host forgets to delete a durable queue in qpid broker associated with the host. That queue is backed up by one such directory pulp.agent.<uuid>, where the <uuid> is the UUID of the unregistered content host.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.