JBoss is unresponsive with threads hanging in HttpURLConnection

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 5.x
  • 6.x
  • 7.x
• Apache Tomcat
  • 6.x
  • 7.x

Issue

• I am using the below code and want to set

HttpURLConnection con = (HttpURLConnection) obj.openConnection();
		String userCredentials = "bpmsAdmin:Admin12345!";
		String basicAuth = "Basic " + new String(new Base64().encode(userCredentials.getBytes()));
		con.setRequestProperty ("Authorization", basicAuth);
		con.setRequestMethod("POST");
		con.setRequestProperty("User-Agent", USER_AGENT);
		con.setRequestProperty("Accept-Language", "en-US,en;q=0.5");

Which code can I use for a rest client (can I use standard java SE code?) that I will also be able to specify timeout?

Resolution

• Try changing the timeouts from the defaults so that these problematic connections are timed out in a timely fashion. When the thread hits this timeout, a SocketTimeoutException will be thrown and the thread will be returned to the pool. You can try changing the timeout defaults through system properties (which are in ms). Add these properties as follows:

If running EAP on Linux:
For EAP4/5 add in $JBOSS_HOME/bin/run.conf and for EAP 6/7 add in $JBOSS_HOME/bin/standalone.conf file.

    JAVA_OPTS="$JAVA_OPTS -Dsun.net.client.defaultConnectTimeout=10000 -Dsun.net.client.defaultReadTimeout=10000"

If running EAP on Windows:
For EAP4/5 add in $JBOSS_HOME/bin/run.conf.bat and for EAP 6/7 add in $JBOSS_HOME/bin/standalone.conf.bat file.

  set "JAVA_OPTS=%JAVA_OPTS%  -Dsun.net.client.defaultConnectTimeout=10000"
  set "JAVA_OPTS=%JAVA_OPTS%  -Dsun.net.client.defaultReadTimeout=10000"

You can also change the timeout programmatically on the HttpURLConnection object directly and handle the resulting SocketTimeoutExceptions from any timeouts as desired.

try {
   HttpURLConnection con = (HttpURLConnection) new URL(url).openConnection();
   con.setConnectTimeout(5000); //set connect timeout to 5 seconds
   con.setReadTimeout(5000); //set read timeout to 5 seconds

...//code using connection
} catch (java.net.SocketTimeoutException e) {
   return false;
} catch (java.io.IOException e) {
   return false;
}

setReadTimeout - Sets the read timeout to a specified timeout, in milliseconds. A non-zero value specifies the timeout when reading from Input stream when a connection is established to a resource. If the timeout expires before there is data available for read, a java.net.SocketTimeoutException is raised. A timeout of zero is interpreted as an infinite timeout.

setConnectTimeout - Sets a specified timeout value, in milliseconds, to be used when opening a communications link to the resource referenced by this URLConnection. If the timeout expires before the connection can be established, a java.net.SocketTimeoutException is raised. A timeout of zero is interpreted as an infinite timeout.

Root Cause

• These stuck threads are handling requests that initiated connections to some third party server.  Communication failed with the third party server but the socket/connection is not being timed out in a timely manner by the default timeout values.  Thus, the thread is stuck on this HttpUrlConnection/Socket.  Threads keep getting stuck until the whole thread pool is stuck, causing server wide unresponsiveness.

Diagnostic Steps

• Troubleshoot using thread dumps and additional steps mentioned in Java application unresponsive

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.