JBoss is unresponsive because the JBossWeb pool is exhausted in socketRead states

Environment

• JBoss Enterprise Application Platform (EAP)
  • 5.x
  • 6.x

Issue

• JBoss becomes unresponsive and thread dumps show the JBossWeb thread pool is at its max with most threads in a socketRead as seen below:
• Sample http thread stack trace:

"http-127.0.0.1-8080-200" daemon prio=10 tid=0x00007f85fc017000 nid=0x22a5 runnable [0x00007f867df9c000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:146)
	at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:729)
	at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:366)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:790)
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:599)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)
	at java.lang.Thread.run(Thread.java:636)

• Sample AJP thread stack trace:

"ajp-/0.0.0.0:8109-721" daemon prio=10 tid=0x00007f2450bb4000 nid=0x68f4 runnable [0x00007f2421b91000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:150)
	at java.net.SocketInputStream.read(SocketInputStream.java:121)
	at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1131)
	at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1213)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:451)
	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:452)
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931)
	at java.lang.Thread.run(Thread.java:722)

Resolution

• Decrease the connectionTimeout of the corresponding connector in your server.xml for EAP 5 and earlier. 10 minutes is usually good for an AJP connector.

      <!-- A HTTP/1.1 Connector on port 8080 -->
      <Connector protocol="HTTP/1.1" port="8080" address="${jboss.bind.address}" 
               connectionTimeout="20000" redirectPort="8443" />

      <!-- A AJP 1.3 Connector on port 8009 -->
      <Connector protocol="AJP/1.3" port="8009" address="${jboss.bind.address}"
         connectionTimeout="600000" redirectPort="8443"/>

• For EAP 6, set this timeout through the following system properties in your standalone or host xml. For example, if you want to set AJP connectionTimeout to 600000 (600000 millseconds = 600 seconds = 10 minutes) and HTTP connectionTimeout to 120000 (120000 millseconds = 120 seconds = 2 minutes):

<system-properties>
    <!-- connectionTimeout for AJP connector. Default value is "-1" (no timeout). -->
    <property name="org.apache.coyote.ajp.DEFAULT_CONNECTION_TIMEOUT" value="600000"/>
    <!-- connectionTimeout for HTTP connector. Default value is "60000". -->
    <property name="org.apache.coyote.http11.DEFAULT_CONNECTION_TIMEOUT" value="120000"/>
</system-properties>

Root Cause

• These threads are in a keepalive state, waiting for another potential request to come on an established connection to a client or proxy.
• AJP by default has no connectionTimeout so may be more prone to this build up if a proxy/loadbalancer is not always heavily reusing its already established connections.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.