Thread leak when performing JMS operations in JBoss EAP

Environment

• JBoss Enterprise Application Platform (EAP)
  • 4.3
  • 5.x
  • 6.x

Issue

• There is an instance of JBoss which frequently sends JMS messages to another instance via the ClusteredConnectionFactory.  Each JMS connection is only used to send one message and then is closed.  However, the thread count in A keeps growing. The thread dump in JBoss A reveals that the majority of the threads are bisocket control threads, with the following stack trace:

    "control: Socket[addr=127.0.0.1,port=45892,localport=51040]" daemon prio=3 tid=0x056c2800 nid=0x2002 runnable [0x5ba4f000]
        java.lang.Thread.State: RUNNABLE
            java.net.SocketInputStream.socketRead0(Native Method)
            java.net.SocketInputStream.read(SocketInputStream.java:129)
            java.net.SocketInputStream.read(SocketInputStream.java:182)
            java.io.FilterInputStream.read(FilterInputStream.java:66)
    org.jboss.remoting.transport.bisocket.BisocketServerInvoker$ControlConnectionThread.run(BisocketServerInvoker.java:748)

• Application performance is very slow and thread dump shows above stack trace.

• Eventually, no new threads can be created in JBoss A and java.lang.OutOfMemoryError: unable to create new native thread errors are received.

Resolution

• Do not open/close a connection using a standard JMS connection factory to send a single message.  This is highly inefficient.  Use a JCA based JMS connection factory instead (e.g. JmsXA) as that will provide pooling.  If for some reason a JCA based connection factory cannot be used then cache the JNDI lookup for the connection factory or do not use the ClusteredConnectionFactory.

Root Cause

• The "extra" control connection is not, in fact, the product of a leak.  It is a special connection used by clustered connection factories and it stays open to inform the client stub about changes in the cluster topology so that when a connection is created using the stub it will be routed to a live node.  This special connection is attached to the connection factory stub so closing the JMS connection won't close it since the stub is still valid and may be used to create additional connections.  The connection will be closed when the stub is finalized.

Diagnostic Steps

• See Java java.lang.OutOfMemoryError "unable to create new native thread" or "Failed to create a thread"
• Check thread dumps for a large number of threads in BisocketServerInvoker$ControlConnectionThread.run
• Verify how JMS connection factories are being used

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.