[Satellite6] Candlepin not responsive with" JournalStorageManager is not responsive"

Solution Verified - Updated 13 Jun 2024

Environment

Red Hat Satellite 6.x

Issue

Journal of candlepin are not responsive. Hammer ping shows candlepin and candlepin_auth as FAIL.

Resolution

Backup old journals and restart candlepin to recreate them.

# foreman-maintain service stop
# mv /var/lib/candlepin/activemq-artemis{,.old}
# foreman-maintain service start
# hammer ping

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

Journals got corrupted or just generally not responsive.

Diagnostic Steps

# hammer ping
candlepin:      
    Status:          FAIL
    Server Response:
candlepin_auth: 
    Status:          FAIL
    Server Response:

cat /var/log/candlepin/error.log

2021-01-01 00:00:01,000 [thread=Thread-0 (-scheduled-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ224081: The component org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager@4b37b721 is not responsive
2021-01-01 00:00:02,000 [thread=Thread-0 (-scheduled-threads)] [=, org=, csid=] WARN  org.apache.activemq.artemis.core.server - AMQ222199: Thread dump: *******************************************************************************
Complete Thread dump 
"IoReceiver - localhost/127.0.0.1:5671" Id=134 RUNNABLE (in native)
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:156)
	at java.lang.Thread.run(Thread.java:748)


"IoSender - localhost/127.0.0.1:5671" Id=135 WAITING on java.lang.Object@3b0234f2
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@3b0234f2
	at java.lang.Object.wait(Object.java:502)
	at org.apache.qpid.transport.network.io.IoSender.run(IoSender.java:279)
	at java.lang.Thread.run(Thread.java:748)


"Thread-18 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1ce4d5e3)" Id=133 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e00bdf6
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e00bdf6
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-17 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1ce4d5e3)" Id=126 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e00bdf6
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e00bdf6
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-16 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1ce4d5e3)" Id=125 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e00bdf6
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e00bdf6
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-15 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1ce4d5e3)" Id=118 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e00bdf6
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e00bdf6
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-14 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$5@1ce4d5e3)" Id=111 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e00bdf6
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e00bdf6
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-4 (ActiveMQ-scheduled-threads)" Id=78 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d65fbfb
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d65fbfb
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-3 (ActiveMQ-scheduled-threads)" Id=77 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d65fbfb
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d65fbfb
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-2 (ActiveMQ-scheduled-threads)" Id=76 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d65fbfb
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d65fbfb
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-1 (ActiveMQ-scheduled-threads)" Id=67 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d65fbfb
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d65fbfb
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Thread-0 (ActiveMQ-scheduled-threads)" Id=66 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d65fbfb
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d65fbfb
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"activemq-buffer-timeout" Id=65 WAITING on java.util.concurrent.Semaphore$NonfairSync@4f34a445
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.Semaphore$NonfairSync@4f34a445
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
	at org.apache.activemq.artemis.core.io.buffer.TimedBuffer$CheckTimer.run(TimedBuffer.java:427)
	at java.lang.Thread.run(Thread.java:748)


"Thread-0 (-scheduled-threads)" Id=64 RUNNABLE
	at sun.management.ThreadImpl.dumpThreads0(Native Method)
	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
	at org.apache.activemq.artemis.utils.ThreadDumpUtil.threadDump(ThreadDumpUtil.java:47)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.threadDump(ActiveMQServerImpl.java:929)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.lambda$initializeCriticalAnalyzer$2(ActiveMQServerImpl.java:642)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$$Lambda$58/402809050.run(Unknown Source)
	at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.fireAction(CriticalAnalyzerImpl.java:145)
	at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.check(CriticalAnalyzerImpl.java:130)
	at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl$1.run(CriticalAnalyzerImpl.java:53)
	at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:284)
	at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$3.run(ActiveMQScheduledComponent.java:294)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@4dce97fc


"pool-3-thread-1" Id=61 RUNNABLE
	at java.lang.Class.getSuperclass(Native Method)
	at java.lang.Enum.getDeclaringClass(Enum.java:199)
	at java.util.EnumSet.of(EnumSet.java:215)
	at sun.security.ssl.Handshaker.getActiveCipherSuites(Handshaker.java:695)
	at sun.security.ssl.Handshaker.activate(Handshaker.java:560)
	at sun.security.ssl.SSLEngineImpl.kickstartHandshake(SSLEngineImpl.java:731)
	-  locked sun.security.ssl.SSLEngineImpl@65bbbcf5
	at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1232)
	-  locked sun.security.ssl.SSLEngineImpl@65bbbcf5
	at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1185)
	-  locked java.lang.Object@1e2f42b9
	at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:471)
	at org.apache.qpid.transport.network.security.ssl.SSLSender.send(SSLSender.java:157)
	at org.apache.qpid.transport.network.security.ssl.SSLSender.send(SSLSender.java:35)
	at org.apache.qpid.transport.network.Disassembler.init(Disassembler.java:160)
	-  locked java.lang.Object@2ca7c8a4
	at org.apache.qpid.transport.network.Disassembler.init(Disassembler.java:48)
	at org.apache.qpid.transport.ProtocolHeader.delegate(ProtocolHeader.java:110)
	at org.apache.qpid.transport.network.Disassembler.send(Disassembler.java:73)
	at org.apache.qpid.transport.network.Disassembler.send(Disassembler.java:48)
	at org.apache.qpid.transport.Connection.send(Connection.java:420)
	at org.apache.qpid.transport.Connection.connect(Connection.java:259)
	-  locked java.lang.Object@46db47f9
	at org.apache.qpid.client.AMQConnectionDelegate_0_10.makeBrokerConnection(AMQConnectionDelegate_0_10.java:222)
	at org.apache.qpid.client.AMQConnection.makeBrokerConnection(AMQConnection.java:664)
	at org.apache.qpid.client.AMQConnection.<init>(AMQConnection.java:444)
	at org.apache.qpid.client.AMQConnectionFactory.createConnection(AMQConnectionFactory.java:124)
	at org.candlepin.audit.QpidConnection.newConnection(QpidConnection.java:183)
	at org.candlepin.audit.QpidQmf.runQuery(QpidQmf.java:101)
	at org.candlepin.audit.QpidQmf.getExchageData(QpidQmf.java:247)
	at org.candlepin.audit.QpidQmf.getStatus(QpidQmf.java:178)
	at org.candlepin.controller.QpidStatusMonitor.monitor(QpidStatusMonitor.java:126)
	-  locked org.candlepin.controller.QpidStatusMonitor@7ebf17c3
	at org.candlepin.controller.QpidStatusMonitor.run(QpidStatusMonitor.java:88)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@10cc74d1


"IoReceiver - localhost/127.0.0.1:5671" Id=59 RUNNABLE (in native)
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:156)
	at java.lang.Thread.run(Thread.java:748)


"IoSender - localhost/127.0.0.1:5671" Id=60 WAITING on java.lang.Object@1a61b827
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@1a61b827
	at java.lang.Object.wait(Object.java:502)
	at org.apache.qpid.transport.network.io.IoSender.run(IoSender.java:279)
	at java.lang.Thread.run(Thread.java:748)


"ack-flusher" Id=54 TIMED_WAITING on java.util.TaskQueue@811b127
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@811b127
	at java.util.TimerThread.mainLoop(Timer.java:552)
	at java.util.TimerThread.run(Timer.java:505)


"QuartzScheduler_QuartzSchedulerThread" Id=51 TIMED_WAITING on java.lang.Object@14e9cb53
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@14e9cb53
	at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:253)


"QuartzScheduler_Worker-15" Id=50 TIMED_WAITING on java.lang.Object@12eed702
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@12eed702
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-14" Id=49 TIMED_WAITING on java.lang.Object@559e5f8
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@559e5f8
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-13" Id=48 TIMED_WAITING on java.lang.Object@c4e4fd
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@c4e4fd
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-12" Id=47 TIMED_WAITING on java.lang.Object@5aad14e5
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@5aad14e5
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-11" Id=46 TIMED_WAITING on java.lang.Object@3e1f8e75
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@3e1f8e75
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-10" Id=45 TIMED_WAITING on java.lang.Object@7423ca6b
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@7423ca6b
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-9" Id=44 TIMED_WAITING on java.lang.Object@53673b1e
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@53673b1e
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-8" Id=43 TIMED_WAITING on java.lang.Object@615f07ab
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@615f07ab
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-7" Id=42 TIMED_WAITING on java.lang.Object@237141d3
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@237141d3
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-6" Id=41 TIMED_WAITING on java.lang.Object@49eba744
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@49eba744
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-5" Id=40 TIMED_WAITING on java.lang.Object@233ecaac
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@233ecaac
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-4" Id=39 TIMED_WAITING on java.lang.Object@3ac5e33f
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@3ac5e33f
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-3" Id=38 TIMED_WAITING on java.lang.Object@1313426d
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@1313426d
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-2" Id=37 TIMED_WAITING on java.lang.Object@75606cc
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@75606cc
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"QuartzScheduler_Worker-1" Id=36 TIMED_WAITING on java.lang.Object@5268c720
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.Object@5268c720
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)


"Statistics Thread-Hibernate 8382b556-d09c-43a5-a323-45e91675db1a-1" Id=34 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47a10a49
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47a10a49
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"Hibernate 8382b556-d09c-43a5-a323-45e91675db1a" Id=33 WAITING on java.util.TaskQueue@4ea828fa
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@4ea828fa
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	at java.util.TimerThread.run(Timer.java:505)


"logback-2" Id=32 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@797f2785
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@797f2785
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"logback-1" Id=31 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@797f2785
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@797f2785
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"C3P0PooledConnectionPoolManager[identityToken->1bqv6yeah1k0kcpmqsik0q|3d3f5cce]-HelperThread-#2" Id=30 TIMED_WAITING on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6d3aae4d
	at java.lang.Object.wait(Native Method)
	-  waiting on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6d3aae4d
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)


"C3P0PooledConnectionPoolManager[identityToken->1bqv6yeah1k0kcpmqsik0q|3d3f5cce]-HelperThread-#1" Id=29 TIMED_WAITING on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6d3aae4d
	at java.lang.Object.wait(Native Method)
	-  waiting on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6d3aae4d
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)


"C3P0PooledConnectionPoolManager[identityToken->1bqv6yeah1k0kcpmqsik0q|3d3f5cce]-HelperThread-#0" Id=28 TIMED_WAITING on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6d3aae4d
	at java.lang.Object.wait(Native Method)
	-  waiting on com.mchange.v2.async.ThreadPoolAsynchronousRunner@6d3aae4d
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)


"C3P0PooledConnectionPoolManager[identityToken->1bqv6yeah1k0kcpmqsik0q|3d3f5cce]-AdminTaskTimer" Id=27 TIMED_WAITING on java.util.TaskQueue@228602e1
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@228602e1
	at java.util.TimerThread.mainLoop(Timer.java:552)
	at java.util.TimerThread.run(Timer.java:505)


"Statistics Thread-file:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ehcache.xml::WebappClassLoader
  context: /candlepin
  delegate: false
  repositories:
    /WEB-INF/classes/
----------> Parent Classloader:
java.net.URLClassLoader@7a81197d
::org.ehcache.jcache.JCacheCachingProvider@78ade268-1" Id=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d6601b3
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d6601b3
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


"file:/var/lib/tomcat/webapps/candlepin/WEB-INF/classes/ehcache.xml::WebappClassLoader
  context: /candlepin
  delegate: false
  repositories:
    /WEB-INF/classes/
----------> Parent Classloader:
java.net.URLClassLoader@7a81197d
::org.ehcache.jcache.JCacheCachingProvider@78ade268" Id=23 WAITING on java.util.TaskQueue@2d255895
	at java.lang.Object.wait(Native Method)
	-  waiting on java.util.TaskQueue@2d255895
	at java.lang.Object.wait(Object.java:502)
	at java.util.TimerThread.mainLoop(Timer.java:526)
	at java.util.TimerThread.run(Timer.java:505)


"localhost-startStop-1" Id=22 RUNNABLE
	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:729)
	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1753)
	-  locked org.apache.activemq.artemis.core.journal.impl.JournalImpl@1555c3ba
	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1390)
	-  locked org.apache.activemq.artemis.core.journal.impl.JournalImpl@1555c3ba
	at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1374)
	at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:846)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2573)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2336)
	-  locked org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl@7564afb
	at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:71)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:543)
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:482)
	-  locked org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl@7564afb
	at org.apache.activemq.artemis.core.server.embedded.EmbeddedActiveMQ.start(EmbeddedActiveMQ.java:115)
	at org.candlepin.audit.ActiveMQContextListener.contextInitialized(ActiveMQContextListener.java:80)
	at org.candlepin.guice.CandlepinContextListener.withInjector(CandlepinContextListener.java:179)
	at org.jboss.resteasy.plugins.guice.GuiceResteasyBootstrapServletContextListener.contextInitialized(GuiceResteasyBootstrapServletContextListener.java:59)
	at org.candlepin.guice.CandlepinContextListener.contextInitialized(CandlepinContextListener.java:140)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5127)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5643)
	-  locked org.apache.catalina.core.StandardContext@6fa44f12
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	-  locked org.apache.catalina.core.StandardContext@6fa44f12
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
	at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
	at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@47c939d0


"Catalina-startStop-1" Id=21 WAITING on java.util.concurrent.FutureTask@1aa0b83b
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.FutureTask@1aa0b83b
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1163)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:497)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1713)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:337)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
	at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:388)
	-  locked org.apache.catalina.core.StandardHost@2e3b5259
	at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:333)
	-  locked org.apache.catalina.core.StandardHost@2e3b5259
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1136)
	-  locked org.apache.catalina.core.StandardHost@2e3b5259
	at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:819)
	-  locked org.apache.catalina.core.StandardHost@2e3b5259
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	-  locked org.apache.catalina.core.StandardHost@2e3b5259
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1571)
	at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1561)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

	Number of locked synchronizers = 1
	- java.util.concurrent.ThreadPoolExecutor$Worker@520a3426


"GC Daemon" Id=20 TIMED_WAITING on sun.misc.GC$LatencyLock@1dc6ba0a
	at java.lang.Object.wait(Native Method)
	-  waiting on sun.misc.GC$LatencyLock@1dc6ba0a
	at sun.misc.GC$Daemon.run(GC.java:117)


"Signal Dispatcher" Id=4 RUNNABLE


"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@26b5e21b
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.ref.ReferenceQueue$Lock@26b5e21b
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)


"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@130175a4
	at java.lang.Object.wait(Native Method)
	-  waiting on java.lang.ref.Reference$Lock@130175a4
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)


"main" Id=1 WAITING on java.util.concurrent.FutureTask@35d9ec15
	at sun.misc.Unsafe.park(Native Method)
	-  waiting on java.util.concurrent.FutureTask@35d9ec15
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:1119)
	-  locked org.apache.catalina.core.StandardEngine@cff669f
	at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:300)
	-  locked org.apache.catalina.core.StandardEngine@cff669f
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	-  locked org.apache.catalina.core.StandardEngine@cff669f
	at org.apache.catalina.core.StandardService.startInternal(StandardService.java:444)
	-  locked org.apache.catalina.core.StandardEngine@cff669f
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	-  locked org.apache.catalina.core.StandardService@24c19be0
	at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:758)
	-  locked java.lang.Object@2ded0059
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
	-  locked org.apache.catalina.core.StandardServer@6adede5
	at org.apache.catalina.startup.Catalina.start(Catalina.java:693)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:428)


===============================================================================
End Thread dump 
*******************************************************************************

2021-01-01 00:01:02,000 [thread=Dispatcher-0-Conn-1031] [=, org=, csid=] ERROR org.apache.qpid.thread.LoggingUncaughtExceptionHandler - Uncaught exception in thread "Dispatcher-0-Conn-1031"
java.lang.IllegalStateException: dispatcher is not started
	at org.apache.qpid.client.AMQSession.dispatch(AMQSession.java:3251)
	at org.apache.qpid.client.message.UnprocessedMessage.dispatch(UnprocessedMessage.java:54)
	at org.apache.qpid.client.AMQSession$Dispatcher.run(AMQSession.java:3403)
	at java.lang.Thread.run(Thread.java:748)

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

candlepin

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.