JWS 5 Tomcat deadlocks across several request threads in java NIO calls

Solution Verified - Updated 13 Jun 2024

Environment

JBoss Web Server (JWS)
- 5.2.0
- 5.3.0

Issue

Tomcat became unresponsive and thread dumps show many request threads blocked in java.nio.channels.spi.AbstractSelector.cancel from a deadlock like below:

Found one Java-level deadlock:
=============================
"http-nio-8080-exec-455":
  waiting to lock monitor 0x00007f59c40475e8 (object 0x00000006c0b4fe28, a java.util.HashSet),
  which is held by "http-nio-8080-ClientPoller"
"http-nio-8080-ClientPoller":
  waiting to lock monitor 0x00007f59680af4d8 (object 0x00000006de810f10, a java.lang.Object),
  which is held by "http-nio-8080-exec-215"
"http-nio-8080-exec-215":
  waiting to lock monitor 0x00007f59c4057a98 (object 0x00000006c0b27028, a java.util.HashSet),
  which is held by "http-nio-8080-BlockPoller"
"http-nio-8080-BlockPoller":
  waiting to lock monitor 0x00007f5998946158 (object 0x00000006de8588c0, a java.lang.Object),
  which is held by "http-nio-8080-exec-378"
"http-nio-8080-exec-378":
  waiting to lock monitor 0x00007f59c40475e8 (object 0x00000006c0b4fe28, a java.util.HashSet),
  which is held by "http-nio-8080-ClientPoller"

Java stack information for the threads listed above:
===================================================
"http-nio-8080-exec-455":
    at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
    - waiting to lock <0x00000006c0b4fe28> (a java.util.HashSet)
    at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
    - locked <0x00000006de870a68> (a sun.nio.ch.SelectionKeyImpl)
    at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
    - locked <0x00000006de870a90> (a java.lang.Object)
    at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
    - locked <0x00000006de870a28> (a java.lang.Object)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:104)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:116)
    at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1179)
    - locked <0x00000006c3ca9be8> (a org.apache.tomcat.util.net.NioChannel)
    at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:394)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:667)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1592)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    - locked <0x00000006de825cc0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
"http-nio-8080-ClientPoller":
    at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
    - waiting to lock <0x00000006de810f10> (a java.lang.Object)
    at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
    at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
    at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
    - locked <0x00000006c0b4fe28> (a java.util.HashSet)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:97)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x00000006c0b50068> (a sun.nio.ch.Util$3)
    - locked <0x00000006c0b50058> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000006c0b4fdd0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:708)
    at java.lang.Thread.run(Thread.java:748)
"http-nio-8080-exec-215":
    at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
    - waiting to lock <0x00000006c0b27028> (a java.util.HashSet)
    at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
    - locked <0x00000006de810c70> (a sun.nio.ch.SelectionKeyImpl)
    at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
    - locked <0x00000006de810f10> (a java.lang.Object)
    at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
    - locked <0x00000006de810c30> (a java.lang.Object)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:104)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:116)
    at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1179)
    - locked <0x00000006c2d7b8a0> (a org.apache.tomcat.util.net.NioChannel)
    at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:394)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:667)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1592)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    - locked <0x00000006de810c98> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
"http-nio-8080-BlockPoller":
    at java.nio.channels.spi.AbstractSelectableChannel.removeKey(AbstractSelectableChannel.java:130)
    - waiting to lock <0x00000006de8588c0> (a java.lang.Object)
    at java.nio.channels.spi.AbstractSelector.deregister(AbstractSelector.java:188)
    at sun.nio.ch.EPollSelectorImpl.implDereg(EPollSelectorImpl.java:191)
    at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:149)
    - locked <0x00000006c0b27028> (a java.util.HashSet)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:90)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
    - locked <0x00000006c0b27108> (a sun.nio.ch.Util$3)
    - locked <0x00000006c0b270f8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x00000006c0b26fc0> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:105)
    at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:310)
"http-nio-8080-exec-378":
    at java.nio.channels.spi.AbstractSelector.cancel(AbstractSelector.java:91)
    - waiting to lock <0x00000006c0b4fe28> (a java.util.HashSet)
    at java.nio.channels.spi.AbstractSelectionKey.cancel(AbstractSelectionKey.java:73)
    - locked <0x00000006de858648> (a sun.nio.ch.SelectionKeyImpl)
    at java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:240)
    - locked <0x00000006de8588c0> (a java.lang.Object)
    at java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
    - locked <0x00000006de8585e0> (a java.lang.Object)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:104)
    at org.apache.tomcat.util.net.NioChannel.close(NioChannel.java:116)
    at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doClose(NioEndpoint.java:1179)
    - locked <0x00000006c3360628> (a org.apache.tomcat.util.net.NioChannel)
    at org.apache.tomcat.util.net.SocketWrapperBase.close(SocketWrapperBase.java:394)
    at org.apache.tomcat.util.net.NioEndpoint$Poller.cancelledKey(NioEndpoint.java:667)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1592)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    - locked <0x00000006de858670> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

Found 1 deadlock.

Resolution

Upgrade to JWS 5.4.0+
As a workaround, you may install JWS native components and configure your connectors to use the APR protocol instead of NIO:

 <Connector port="8080"
           protocol="org.apache.coyote.http11.Http11AprProtocol" ...

Root Cause

Content from bz.apache.org is not included.ASF BZ-64007

SBR

Product(s)

Red Hat JBoss Web Server

Components

tomcat9

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.