JBoss has many waiting ConnectionTable.Connection.Sender threads

Environment

• JBoss Enterprise Application Platform (EAP) 5

Issue

• JBoss is filling with thousands of waiting ConnectionTable.Connection.Sender threads, for example

"ConnectionTable.Connection.Sender local_addr=127.0.0.1:7803 [127.0.0.1:7803 - 127.0.0.1:48320],prod-partition,127.0.0.1:7803" - Thread t@81646
   java.lang.Thread.State: WAITING
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for <45594cbf> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
	at org.jgroups.blocks.BasicConnectionTable$Connection$Sender.run(BasicConnectionTable.java:753)
	at java.lang.Thread.run(Thread.java:662)

Resolution

• Address any underlying network issues breaking the cluster connections

Root Cause

• A network issue broke cluster connections repeatedly. There is one of those threads for each connection to another JGroups instance so broken connections can cause new connections and new threads.

Diagnostic Steps

• Capture lsof and netstat -vatnp output along with thread dumps to check thread and socket counts and states

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.