Satellite6 performance issues every day after noon

Solution Verified - Updated

Environment

Red Hat Satellite 6

Issue

  • subscription related requests (i.e. against candlepin) take significantly longer
  • candlepin consumes much CPU
  • after some random time (e.g. several tens of minutes), the problems disappear
  • the issue usually starts to happen every day after noon
  • the problems are getting worse over days, usually - tomcat restart alleviates the problems to some extent

Resolution

Ensure first the diagnostic steps match with your symptoms. If the problem is really CertificateRevocationListTask (CRL task), then as a workaround, disable the task for a long time until a resolution fix is available:

  • add to /etc/candlepin/candlepin.conf :
pinsetter.org.candlepin.pinsetter.tasks.CertificateRevocationListTask.schedule=0 0 0 1 1 ?
  • restart candlepin:
systemctl restart tomcat.service

Be aware, the above change will be overwritten by running installer or Satellite upgrade. Please contact Red Hat support in case you need to make the change more permanent.

For a permanent solution, wait until the This content is not included.improvement to disable the CRL by installer is implemented.

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Diagnostic Steps

/var/log/candlepin/candlepin.logcontains logs about starting CertificateRevocationListTask:

2019-02-05 12:00:00,104 [thread=QuartzScheduler_Worker-10] [job=CertificateRevocationListTask-f44c921f-8dc8-4928-ace1-3ebd9fb31f0c, org=, csid=] INFO  org.candlepin.pinsetter.tasks.KingpinJob - Starting job: org.candlepin.pinsetter.tasks.CertificateRevocationListTask
2019-02-05 12:00:00,104 [thread=QuartzScheduler_Worker-10] [job=CertificateRevocationListTask-f44c921f-8dc8-4928-ace1-3ebd9fb31f0c, org=, csid=] INFO  org.candlepin.pinsetter.tasks.CertificateRevocationListTask - Executing CRL Job. CRL filePath=/var/lib/candlepin/candlepin-crl.crl

but not completion of the task - log record like below is missing:

2019-02-05 12:00:00,112 [thread=QuartzScheduler_Worker-10] [job=CertificateRevocationListTask-f44c921f-8dc8-4928-ace1-3ebd9fb31f0c, org=, csid=] INFO  org.candlepin.pinsetter.tasks.KingpinJob - Job completed: time=8

Optionally, the job execution takes a huge time (more than few minutes), like:

2019-12-10 12:00:03,938 [thread=QuartzScheduler_Worker-7] [job=CertificateRevocationListTask-7f493c61-33e5-46f1-9120-6a962f6c1eea, org=Default_Organization, csid=] INFO  org.candlepin.pinsetter.tasks.KingpinJob - Starting job: org.candlepin.pinsetter.tasks.CertificateRevocationListTask
2019-12-10 12:00:03,938 [thread=QuartzScheduler_Worker-7] [job=CertificateRevocationListTask-7f493c61-33e5-46f1-9120-6a962f6c1eea, org=Default_Organization, csid=] INFO  org.candlepin.pinsetter.tasks.CertificateRevocationListTask - Executing CRL Job. CRL filePath=/var/lib/candlepin/candlepin-crl.crl
2019-12-10 22:56:39,774 [thread=QuartzScheduler_Worker-5] [job=CertificateRevocationListTask-7f493c61-33e5-46f1-9120-6a962f6c1eea, org=, csid=] INFO  org.candlepin.pinsetter.tasks.KingpinJob - Job completed: time=125799707
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.