HTTP Connection is being closed by EAP when sending requests with large body

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 6.4.7 +

Issue

After upgrade to JBoss Enterprise Application Platform (EAP) 6.4.7+, we observe the issue that when sending requests with large body (>2MB), connections are closed by EAP unexpectedly.

Here is the error we get from .Net client side:

"2016-07-22 16:40:06:30","Error","AdaptorBase:ExecuteImpl()","Exception occured: System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host at System.Net.Sockets.Socket.MultipleSend(BufferOffsetSize[] buffers, SocketFlags socketFlags) at ......

• CVE-2014-0230 - Non-persistent DoS attack by feeding data by aborting an upload

Resolution

Starting from EAP 6.4.7, with CVE-2014-0230 fixed, for those requests > 2MB (configurable), after returning the response, connection will be killed / aborted.

System property org.apache.coyote.MAX_SWALLOW_SIZE (unit is byte, default value is 2097152, e.g. 2MB) can be configured to adjust this threshold. For example, you can set 20MB as Java VM option like:

./bin/standalone.sh  ...-Dorg.apache.coyote.MAX_SWALLOW_SIZE=20971520 ...

or you can set it in standalone.xml/domain.xml like the following. Please also refer to How to add/remove/update system properties in EAP 6 for details about CLI command:

    <system-properties>
        <property name="org.apache.coyote.MAX_SWALLOW_SIZE" value="20971520"/>
    <system-properties>

Explanation to org.apache.coyote.MAX_SWALLOW_SIZE:

Limits the length of a request body JBoss/Tomcat will swallow if it is not read during normal request processing. If the value is -1, no limit will be imposed.
If not specified, the default value of 2097152 (2MB) will be used.

More importantly, please review and fix your client code and make sure connection is not used anymore if a response is returned and closed by remote host.

Root Cause

• .Net client application keeps an old connection and reuse it after it's closed by remote host (JBoss)

• This CVE-2014-0230 is fixed on EAP 6.4.7 as security advisory This content is not included.RHSA-2016:0599-1, effectively killing the connection and not reading any more of the data, given the response has been returned. Here's the explanation of CVE-2014-0230:

  When a response for a request with a request body is returned to the user agent before the request body is fully read, by default JBoss/Tomcat swallows the remaining request body so that the next request on the connection may be processed. There was no limit to the size of request body that JBoss/Tomcat would swallow. This permitted a limited Denial of Service as JBoss/Tomcat would never close the connection and a processing thread would remain allocated to the connection.
  

  With fixing this vulnerability, JBoss/Tomcat will kill the connection after returning the response when receiving requests larger than org.apache.coyote.MAX_SWALLOW_SIZE (= maxSwallowSize, default value is 2097152, e.g. 2MB).

Diagnostic Steps

• When you are using EAP 6.4. CP14 (6.4.14) which incorporates This content is not included.BZ#1399005 or later, you can know if your are hitting this org.apache.coyote.MAX_SWALLOW_SIZE (= maxSwallowSize) setting or not by enabling DEBUG logging for org.apache.coyote.http11 category. It will output DEBUG level logging message "Maximum swallow size XXXX is exceeded".

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.