AsyncContext.complete() blocks sometimes as long as the DEFAULT_CONNECTION_TIMEOUT

Environment

• JBoss Enterprise Application Platform (EAP) 6.4.x
  • 6.4.11
  • 6.4.12
  • 6.4.13

Issue

• For a small number of asynchronous http requests after updating 6.4.11+, invoking AsyncContext.complete() blocks the current thread, sometimes for as long as the timeout set in "org.apache.coyote.http11.DEFAULT_CONNECTION_TIMEOUT". For instance:

"pool-5-thread-3" prio=6 tid=0x000000000b3f9800 nid=0x11f0 waiting for monitor entry [0x000000000e5bf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:1149)
        - waiting to lock <0x00000000eb88a8b8> (a org.apache.coyote.http11.Http11Processor)
        at org.apache.coyote.Request.action(Request.java:360)
        at org.apache.catalina.connector.Request.wakeup(Request.java:2601)
        at org.apache.catalina.connector.Request$AsyncContextImpl.complete(Request.java:3419)
        at async.servlet.AsyncReadServlet$1.run(AsyncReadServlet.java:69)
        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.util.concurrent.FutureTask.run(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(Unknown Source)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

   Locked ownable synchronizers:
        - <0x00000000eda09618> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"http-127.0.0.1:8080-2" daemon prio=6 tid=0x000000000b3f3800 nid=0x22b0 runnable [0x000000000df3f000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:713)
        at org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:351)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:819)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:656)
        - locked <0x00000000eb88a8b8> (a org.apache.coyote.http11.Http11Processor)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:921)
        at java.lang.Thread.run(Unknown Source)

Resolution

• Use the NIO connector instead of JIO as a workaround:

<connector name="http" protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="http" socket-binding="http" />

• Upgrade to 6.4.15 when available.

Root Cause

• This content is not included.BZ-1410869
  • This blockage is introduced by the synchronization added to avoid race conditions leading to This content is not included.BZ-1370182
  • A timing issue allows for a request processor to be improperly reused across multiple request, allowing for an aysnc call to block on a separate unrelated request attempt

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.