Satellite 6: candlepin and candlepin_auth fail with response "404 Resource Not Found" and "Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found" (GC overhead limit exceeded)

Solution Verified - Updated

Environment

  • Red Hat Satellite 6

Issue

  • In Satellite 6 WebUI the Content Hosts cannot be loaded and the loading icon does not disappear. When checking status of services via hammer command, candlepin and candlepin_auth fail with response "404 Resource Not Found".

Resolution

Edit "/etc/sysconfig/tomcat" and insert the appropriate value for minimum (-Xms) and maximum heap (-Xmx) size:

JAVA_OPTS="-Xms1024m -Xmx8192m"

After that restart tomcat service:

# service tomcat restart

Check the status of Candlepin via "hammer ping" .

Often the cause of Candlepin high memory requirement is "virt-who" service which has been configured to report to Satellite too frequently. If you are experiencing these problems, make sure that the VIRTWHO_INTERVAL in /etc/sysconfig/virt-who is at least 1800 seconds. After this has been corrected, go to the Satellite server and delete all files in "/var/lib/candlepin/hornetq/journal" and "/var/lib/candlepin/hornetq/largemsgs" directories. Subsequently, reduce the required heap size in "/etc/sysconfig/tomcat".

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

The cause of OutofMemory error is when Tomcat uses up all of the heap memory allocated to it. Heap memory is a dynamically growing and shrinking block of memory where Java objects reside.

Diagnostic Steps

"hammer ping" shows the following status:

# hammer ping
candlepin:      
    Status:          FAIL
    Server Response: Message: 404 Resource Not Found
candlepin_auth: 
    Status:          FAIL
    Server Response: Message: Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found  (GET /candlepin/status)
pulp:           
    Status:          ok
    Server Response: Duration: 224ms
pulp_auth:      
    Status:          ok
    Server Response: Duration: 19ms
elasticsearch:  
    Status:          ok
    Server Response: Duration: 11ms
foreman_tasks:  
    Status:          ok
    Server Response: Duration: 1ms

In /var/log/tomcat/catalina.YEAR-MONTH-DAY.log the following messages are created:

SEVERE: Error waiting for multi-thread deployment of directories to complete
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1150)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:490)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1614)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:330)
        at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
        at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:402)
        at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:347)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1140)
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:799)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.