Candlepin gets stuck during startup forever, logging huge thread dump to error.log

Solution Verified - Updated 14 Jun 2024

Environment

Red Hat Satellite 6.4 or newer

Issue

having problems to send messages to qpidd in past, such that candlepin was in Suspended mode
resolving the issues with qpidd broker / katello_event_queue / ListenOnCandlepinEvents task
restarting tomcat to restore candlepin fails after few minutes, logging huge thread dump to /var/log/candlepin/error.log
candlepin CPU utilization is extremely high, such as a few hundreds percent to thousands percent and all the "/rhsm" related passenger requests are stuck and queuing for extremely long time, such as 30 minutes to an hour.

Resolution

Upgrade to Red Hat Satellite 6.7.2 with Errata RHBA-2020:3255

Or, pre-6.7.2 workaround: When tomcat is stopped, remove (or potentially backup) content of /var/lib/candlepin/activemq-artemis:

  # systemctl stop tomcat
  # rm -rf /var/lib/candlepin/activemq-artemis/bindings/*
  # rm -rf /var/lib/candlepin/activemq-artemis/journal/*
  # rm -rf /var/lib/candlepin/activemq-artemis/paging/*
  # systemctl start tomcat

Now candlepin will start up properly. Since some candlepin events were deleted by the procedure and katello is not aware of them, it should be populated by reimporting subscriptions and pools, via:
```
  # foreman-rake katello:import_subscriptions
```

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

When candlepin cant send events as messages to qpidd broker (i.e. due to ListenOnCandlepinEvents task down, katello_event_queue full refusing further messages from candlepin), it buffers the messages in its internal broker. Since Satellite 6.4, the broker is Content from activemq.apache.org is not included.ActiveMQ Artemis.
Satellite 6.6 AND ABOVE ONLY: Candlepin dropped the event.org.candlepin.audit.DatabaseListener but the queue is still remain in the internal broker (Content from activemq.apache.org is not included.ActiveMQ Artemis ). Candlepin is still sending the event messages to the queue but there is no more DatabaseListener to consume the messages so the internal broker will be flooded with the messages after some time as they will never get deleted. Artemis broker will write the messages to disk (paging) when the size of the messages in the queue exceeded the configured size (10MB) which will eventually fill up the disk space.
A This content is not included.bug in candlepin that prevents candlepin from startup when the internal broker keeps too many messages. To restore service, the buffered messages must be purged away, by removing content of the /var/lib/candlepin/activemq-artemis/journal/ directory. A fix for that bug was released with Red Hat Satellite 6.7.2 errata RHBA-2020:3255

Diagnostic Steps

/var/lib/candlepin/activemq-artemis directory size is large

  # du -sh /var/lib/candlepin/activemq-artemis
  7.0G	/var/lib/candlepin/activemq-artemis

Satellite 6.6 AND ABOVE ONLY: If you see a warning regarding event.org.candlepin.audit.DatabaseListener queue in /var/log/candlepin/error.log a few minutes after Candlepin startup:

  2020-01-08 09:42:16,732 [thread=Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@47a86b09)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222165: No Dead Letter Address configured for queue event.org.candlepin.audit.DatabaseListener in AddressSettings
  2020-01-08 09:42:16,732 [thread=Thread-23 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@47a86b09)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222166: No Expiry Address configured for queue event.org.candlepin.audit.DatabaseListener in AddressSettings

/var/log/candlepin/error.log logs few minutes after startup:

  2019-01-14 12:28:14,415 [thread=Thread-0 (-scheduled-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222199: Thread dump: *******************************************************************************
  Complete Thread dump 
  "Thread-4 (ActiveMQ-scheduled-threads)" Id=63 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
  	at sun.misc.Unsafe.park(Native Method)
  	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
  	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)


  "Thread-3 (ActiveMQ-scheduled-threads)" Id=62 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
  	at sun.misc.Unsafe.park(Native Method)
  	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
  	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)


  "Thread-2 (ActiveMQ-scheduled-threads)" Id=61 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
  	at sun.misc.Unsafe.park(Native Method)
  	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
  	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)


  "Thread-1 (ActiveMQ-scheduled-threads)" Id=60 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
  	at sun.misc.Unsafe.park(Native Method)
  	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
  	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)


  "Thread-0 (ActiveMQ-scheduled-threads)" Id=57 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
  	at sun.misc.Unsafe.park(Native Method)
  	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c00f6ce
  	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)


  "activemq-buffer-timeout" Id=56 WAITING on java.util.concurrent.Semaphore$NonfairSync@3902672a
  	at sun.misc.Unsafe.park(Native Method)
  	-  waiting on java.util.concurrent.Semaphore$NonfairSync@3902672a
  	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
  	at org.apache.activemq.artemis.core.io.buffer.TimedBuffer$CheckTimer.run(TimedBuffer.java:427)
  	at java.lang.Thread.run(Thread.java:748)


  "Thread-0 (-scheduled-threads)" Id=55 RUNNABLE
  	at sun.management.ThreadImpl.dumpThreads0(Native Method)
  	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
  	at org.apache.activemq.artemis.utils.ThreadDumpUtil.threadDump(ThreadDumpUtil.java:47)
  	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.threadDump(ActiveMQServerImpl.java:929)
  	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.lambda$initializeCriticalAnalyzer$2(ActiveMQServerImpl.java:642)
  	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$$Lambda$13/1979890430.run(Unknown Source)
  	at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.fireAction(CriticalAnalyzerImpl.java:145)
  	at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.check(CriticalAnalyzerImpl.java:130)
  	at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl$1.run(CriticalAnalyzerImpl.java:53)
  	at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:284)
  	at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$3.run(ActiveMQScheduledComponent.java:294)
  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)

  	Number of locked synchronizers = 1
  	- java.util.concurrent.ThreadPoolExecutor$Worker@4a914d3d


  "pool-2-thread-1" Id=52 RUNNABLE
  	at java.io.FileInputStream.readBytes(Native Method)
  	at java.io.FileInputStream.read(FileInputStream.java:255)
  	at sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:424)
  	at sun.security.provider.NativePRNG$RandomIO.ensureBufferValid(NativePRNG.java:525)
  	at sun.security.provider.NativePRNG$RandomIO.implNextBytes(NativePRNG.java:544)
  	-  locked java.lang.Object@ba508d9
  	at sun.security.provider.NativePRNG$RandomIO.access$400(NativePRNG.java:331)
  	at sun.security.provider.NativePRNG.engineNextBytes(NativePRNG.java:220)
  	at java.security.SecureRandom.nextBytes(SecureRandom.java:468)
  	at java.security.SecureRandom.next(SecureRandom.java:491)
  	at java.util.Random.nextInt(Random.java:329)
  	at sun.security.ssl.SSLContextImpl.engineInit(SSLContextImpl.java:101)
  	at javax.net.ssl.SSLContext.init(SSLContext.java:282)
  	at org.apache.qpid.ssl.SSLContextFactory.buildContext(SSLContextFactory.java:131)
  	at org.apache.qpid.ssl.SSLContextFactory.buildClientContext(SSLContextFactory.java:62)
  	at org.apache.qpid.transport.network.security.SecurityLayerFactory$SSLSecurityLayer.<init>(SecurityLayerFactory.java:79)
  	at org.apache.qpid.transport.network.security.SecurityLayerFactory.newInstance(SecurityLayerFactory.java:52)
  	at org.apache.qpid.transport.Connection.connect(Connection.java:233)
  	-  locked java.lang.Object@8f1bb59
  	at org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:222)
  	at org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:664)
  	at org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:444)
  	at org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124)
  	at org.candlepin.audit.QpidConnection.newConnection(QpidConnection.java:170)
  	at org.candlepin.audit.QpidQmf.runQuery(QpidQmf.java:101)
  	at org.candlepin.audit.QpidQmf.getExchangeBoundQueueNames(QpidQmf.java:220)
  	at org.candlepin.audit.QpidQmf.getStatus(QpidQmf.java:178)
  	at org.candlepin.controller.QpidStatusMonitor.monitor(QpidStatusMonitor.java:124)
  	-  locked org.candlepin.controller.QpidStatusMonitor@73bc8a0f
  	at org.candlepin.controller.QpidStatusMonitor.run(QpidStatusMonitor.java:86)
  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)

  	Number of locked synchronizers = 1
  	- java.util.concurrent.ThreadPoolExecutor$Worker@3db5f5a6


  "IoReceiver - localhost/127.0.0.1:5671" Id=50 RUNNABLE (in native)
  	at java.net.SocketInputStream.socketRead0(Native Method)
  	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
  	at java.net.SocketInputStream.read(SocketInputStream.java:171)
  	at java.net.SocketInputStream.read(SocketInputStream.java:141)
  	at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:156)
  	at java.lang.Thread.run(Thread.java:748)


  "IoSender - localhost/127.0.0.1:5671" Id=51 WAITING on java.lang.Object@592d9865
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@592d9865
  	at java.lang.Object.wait(Object.java:502)
  	at org.apache.qpid.transport.network.io.IoSender.run(IoSender.java:279)
  	at java.lang.Thread.run(Thread.java:748)


  "ack-flusher" Id=45 TIMED_WAITING on java.util.TaskQueue@7aedb059
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.util.TaskQueue@7aedb059
  	at java.util.TimerThread.mainLoop(Timer.java:552)
  	at java.util.TimerThread.run(Timer.java:505)


  "QuartzScheduler_QuartzSchedulerThread" Id=42 TIMED_WAITING on java.lang.Object@7f43e72e
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@7f43e72e
  	at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:253)


  "QuartzScheduler_Worker-15" Id=41 TIMED_WAITING on java.lang.Object@68b8e63b
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@68b8e63b
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-14" Id=40 TIMED_WAITING on java.lang.Object@5d63a5f9
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@5d63a5f9
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-13" Id=39 TIMED_WAITING on java.lang.Object@4a1ba974
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@4a1ba974
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-12" Id=38 TIMED_WAITING on java.lang.Object@2a815d8e
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@2a815d8e
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-11" Id=37 TIMED_WAITING on java.lang.Object@527efdb
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@527efdb
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-10" Id=36 TIMED_WAITING on java.lang.Object@45abceeb
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@45abceeb
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-9" Id=35 TIMED_WAITING on java.lang.Object@52ec8f23
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@52ec8f23
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-8" Id=34 TIMED_WAITING on java.lang.Object@72a1d5dd
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@72a1d5dd
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-7" Id=33 TIMED_WAITING on java.lang.Object@48c2a565
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@48c2a565
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-6" Id=32 TIMED_WAITING on java.lang.Object@5385f600
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@5385f600
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-5" Id=31 TIMED_WAITING on java.lang.Object@65a6fec4
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@65a6fec4
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-4" Id=30 TIMED_WAITING on java.lang.Object@1955d090
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@1955d090
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-3" Id=29 TIMED_WAITING on java.lang.Object@4461b752
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@4461b752
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-2" Id=28 TIMED_WAITING on java.lang.Object@38286c28
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@38286c28
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "QuartzScheduler_Worker-1" Id=27 TIMED_WAITING on java.lang.Object@6aaaf537
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.Object@6aaaf537
  	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


  "Statistics Thread-__DEFAULT__-1" Id=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5dcc79a2
  	at sun.misc.Unsafe.park(Native Method)
  	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5dcc79a2
  	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)


  "__DEFAULT__" Id=24 WAITING on java.util.TaskQueue@14de9e6e
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.util.TaskQueue@14de9e6e
  	at java.lang.Object.wait(Object.java:502)
  	at java.util.TimerThread.mainLoop(Timer.java:526)
  	at java.util.TimerThread.run(Timer.java:505)


  "C3P0PooledConnectionPoolManager[identityToken->2tmgiaa0l499gq1p2apom|33acba40]-HelperThread-#2" Id=23 TIMED_WAITING on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
  	at java.lang.Object.wait(Native Method)
  	-  waiting on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
  	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)


  "C3P0PooledConnectionPoolManager[identityToken->2tmgiaa0l499gq1p2apom|33acba40]-HelperThread-#1" Id=22 TIMED_WAITING on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
  	at java.lang.Object.wait(Native Method)
  	-  waiting on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
  	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)


  "C3P0PooledConnectionPoolManager[identityToken->2tmgiaa0l499gq1p2apom|33acba40]-HelperThread-#0" Id=21 TIMED_WAITING on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
  	at java.lang.Object.wait(Native Method)
  	-  waiting on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6f6cf9f1
  	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)


  "C3P0PooledConnectionPoolManager[identityToken->2tmgiaa0l499gq1p2apom|33acba40]-AdminTaskTimer" Id=20 TIMED_WAITING on java.util.TaskQueue@5252aa66
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.util.TaskQueue@5252aa66
  	at java.util.TimerThread.mainLoop(Timer.java:552)
  	at java.util.TimerThread.run(Timer.java:505)


  "Statistics Thread-file:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ehcache.xml::WebappClassLoader
    context: /candlepin
    delegate: false
    repositories:
      /WEB-INF/classes/
  ----------> Parent Classloader:
  java.net.URLClassLoader@7a81197d
  ::org.ehcache.jcache.JCacheCachingProvider@2bdd47be-1" Id=17 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6cedd8e7
  	at sun.misc.Unsafe.park(Native Method)
  	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6cedd8e7
  	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
  	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
  	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)


  "file:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ehcache.xml::WebappClassLoader
    context: /candlepin
    delegate: false
    repositories:
      /WEB-INF/classes/
  ----------> Parent Classloader:
  java.net.URLClassLoader@7a81197d
  ::org.ehcache.jcache.JCacheCachingProvider@2bdd47be" Id=16 WAITING on java.util.TaskQueue@c2016fa
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.util.TaskQueue@c2016fa
  	at java.lang.Object.wait(Object.java:502)
  	at java.util.TimerThread.mainLoop(Timer.java:526)
  	at java.util.TimerThread.run(Timer.java:505)


  "com.google.inject.internal.util.$Finalizer" Id=15 WAITING on java.lang.ref.ReferenceQueue$Lock@40bd5fe
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.ref.ReferenceQueue$Lock@40bd5fe
  	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
  	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
  	at com.google.inject.internal.util.$Finalizer.run(Finalizer.java:114)


  "localhost-startStop-1" Id=14 RUNNABLE
  	at org.apache.activemq.artemis.core.journal.impl.JournalImpl$11.checkDeleteSize(JournalImpl.java:1406)
  	at org.apache.activemq.artemis.core.journal.impl.JournalImpl$11.deleteRecord(JournalImpl.java:1435)
  	at org.apache.activemq.artemis.core.journal.impl.JournalImpl$13.onReadCommitRecord(JournalImpl.java:1932)
  	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:729)
  	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1753)
  	-  locked org.apache.activemq.artemis.core.journal.impl.JournalImpl@434d4e83
  	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1390)
  	-  locked org.apache.activemq.artemis.core.journal.impl.JournalImpl@434d4e83
  	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1374)
  	at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:846)
  	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2573)
  	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2336)
  	-  locked org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl@41658ced
  	at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:71)
  	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:543)
  	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:482)
  	-  locked org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl@41658ced
  	at org.apache.activemq.artemis.core.server.embedded.EmbeddedActiveMQ.start(EmbeddedActiveMQ.java:115)
  	at org.candlepin.audit.ActiveMQContextListener.contextInitialized(ActiveMQContextListener.java:159)
  	at org.candlepin.guice.CandlepinContextListener.withInjector(CandlepinContextListener.java:182)
  	at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextInitialized(GuiceResteasyBootstrapServletContextListener.java:57)
  	at org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:144)
  	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5118)
  	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634)
  	-  locked org.apache.catalina.core.StandardContext@6939ae95
  	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
  	-  locked org.apache.catalina.core.StandardContext@6939ae95
  	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
  	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
  	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
  	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
  	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)

  	Number of locked synchronizers = 1
  	- java.util.concurrent.ThreadPoolExecutor$Worker@5541d560


  "Catalina-startStop-1" Id=13 WAITING on java.util.concurrent.FutureTask@4d7ed5fd
  	at sun.misc.Unsafe.park(Native Method)
  	-  waiting on java.util.concurrent.FutureTask@4d7ed5fd
  	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
  	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
  	at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1163)
  	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:497)
  	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1713)
  	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:337)
  	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
  	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
  	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:388)
  	-  locked org.apache.catalina.core.StandardHost@26024709
  	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:333)
  	-  locked org.apache.catalina.core.StandardHost@26024709
  	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1136)
  	-  locked org.apache.catalina.core.StandardHost@26024709
  	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:819)
  	-  locked org.apache.catalina.core.StandardHost@26024709
  	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
  	-  locked org.apache.catalina.core.StandardHost@26024709
  	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1571)
  	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1561)
  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)

  	Number of locked synchronizers = 1
  	- java.util.concurrent.ThreadPoolExecutor$Worker@42dafa95


  "GC Daemon" Id=12 TIMED_WAITING on sun.misc.GC$LatencyLock@17804d89
  	at java.lang.Object.wait(Native Method)
  	-  waiting on sun.misc.GC$LatencyLock@17804d89
  	at sun.misc.GC$Daemon.run(GC.java:117)


  "Signal Dispatcher" Id=4 RUNNABLE


  "Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3e9030ac
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.ref.ReferenceQueue$Lock@3e9030ac
  	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
  	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
  	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)


  "Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@53ffc177
  	at java.lang.Object.wait(Native Method)
  	-  waiting on java.lang.ref.Reference$Lock@53ffc177
  	at java.lang.Object.wait(Object.java:502)
  	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
  	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)


  "main" Id=1 WAITING on java.util.concurrent.FutureTask@6f9f0c51
  	at sun.misc.Unsafe.park(Native Method)
  	-  waiting on java.util.concurrent.FutureTask@6f9f0c51
  	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
  	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
  	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1119)
  	-  locked org.apache.catalina.core.StandardEngine@681537c8
  	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:300)
  	-  locked org.apache.catalina.core.StandardEngine@681537c8
  	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
  	-  locked org.apache.catalina.core.StandardEngine@681537c8
  	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:444)
  	-  locked org.apache.catalina.core.StandardEngine@681537c8
  	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
  	-  locked org.apache.catalina.core.StandardService@7d6cc346
  	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:758)
  	-  locked java.lang.Object@42468917
  	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
  	-  locked org.apache.catalina.core.StandardServer@3e6fa38a
  	at org.apache.catalina.startup.Catalina.start(Catalina.java:693)
  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.lang.reflect.Method.invoke(Method.java:498)
  	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
  	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:428)


  ===============================================================================
  End Thread dump 
  *******************************************************************************

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

candlepin

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.