[Satellite6] Candlepin request takes a much longer time than it should

Solution Verified - Updated

Environment

Red Hat Satellite 6.2

Issue

Candlepin or postgres high CPU usage
Candlepin requests take a much longer time than they should (a few minutes rather than a few seconds)
Some candlepin requests fail with a '503 service unavailable' error

Resolution

Workaround: add an index to make searching in postgres faster

su - postgres -c "psql candlepin -c \"create index cp_consumer_fact_mapkey_idx on cp_consumer_facts (mapkey, element) where mapkey='virt.uuid';\""

Solution

This content is not included.Bugzilla 1554421

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

When there are a few thousand records in cp_consumer table and approximatively 1 000 000 records in cp_consumer_facts is, candlepin requests take a few minutes instead of a few seconds.

Diagnostic Steps

Check the records count in cp_consumer and cp_consumer_facts

su - postgres -c "psql candlepin -c \"select count(*) from cp_consumer;\""
su - postgres -c "psql candlepin -c \"select count(*) from cp_consumer_facts;\""

Check the requests time (results in milliseconds)

grep /var/log/candlepin/candlepin.log -v -e COMPLIANCE -e ENTITLEMENT | grep LoggingFilter | grep time | awk '{print $13}' | cut -d= -f2 | sort -n

Based on the previous outputs, take the longest time and find if the request type is /candlepin/consumers//guests

grep `grep <time> /var/log/candlepin/candlepin.log | awk '{print $4}' | cut -d= -f2 | cut -d, -f1` /var/log/candlepin/candlepin.log  | grep uri

Apply Solution 2955931
You can also manually check how much time the request takes

time curl -u admin:admin -ks https://localhost:8443/<uri from previous command>
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.