EAP 6 shutdown stalls when using an Executor pool for JBossWeb

Solution Verified - Updated 6 Aug 2024

Environment

JBoss Enterprise Application Platform (EAP) 6

Issue

EAP 6 fails to shutdown fully when using an Executor pool for a JBossWeb connector.
Thread dumps show shutdown threads waiting like so:

"MSC Shutdown Thread" prio=6 tid=0x5552d000 nid=0x208c waiting on condition [0x55a3f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x13bf0000> (a java.util.concurrent.CountDownLatch$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
	at org.jboss.msc.service.ServiceContainerImpl$ShutdownHookHolder$1$1.run(ServiceContainerImpl.java:129)
	at java.lang.Thread.run(Thread.java:662)

"SIGINT handler" daemon prio=10 tid=0x5552c400 nid=0x6f8 in Object.wait() [0x5599f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x181732f8> (a java.lang.Thread)
	at java.lang.Thread.join(Thread.java:1186)
	- locked <0x181732f8> (a java.lang.Thread)
	at java.lang.Thread.join(Thread.java:1239)
	at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
	at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
	at java.lang.Shutdown.runHooks(Shutdown.java:79)
	at java.lang.Shutdown.sequence(Shutdown.java:123)
	at java.lang.Shutdown.exit(Shutdown.java:168)
	- locked <0x42fe8158> (a java.lang.Class for java.lang.Shutdown)
	at java.lang.Terminator$1.handle(Terminator.java:35)
	at sun.misc.Signal$1.run(Signal.java:195)
	at java.lang.Thread.run(Thread.java:662)

There are also many threads in a socket read during their keepalive state:

"http-executor-threads - 1" prio=6 tid=0x55515000 nid=0x20c0 runnable [0x5590f000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1131)
	at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1213)
	at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:451)
	at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:452)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:519)
	at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
	at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:801)
	at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
	at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:821)
	at java.lang.Thread.run(Thread.java:662)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

When we shutdown the proxy/loadbalancer end for those AJP connections, those threads clear out and then shutdown completes.

Resolution

Do not use an executor with JBossWeb
To decrease the shutdown wait time when using an executor pool with AJP connections, lower the org.apache.coyote.ajp.DEFAULT_CONNECTION_TIMEOUT system property. This is infinite by default, meaning keepalive AJP connections would never timeout and the executor threads would never finish to allow shutdown.
For a HTTP(S) connector, set org.apache.coyote.http11.DEFAULT_CONNECTION_TIMEOUT (default 60000 ms)
Address any long running or stalled requests.

Root Cause

The Executor waits for its threads to complete before it allows shutdown to proceed. It tries to interrupt them to ensure that happens quickly but the interrupt does nothing to the keepalive socket reads. Also, there is no guarantee that the interrupt can end all processing requests, and the executor cannot shutdown while a request processes.
AJP connections typically have long keepalive states and so this can stall shutdown for quite some time.
This content is not included.Bug 882048

Diagnostic Steps

Troubleshoot using thread dumps, garbage collection logging, and additional steps as described in Java application unresponsive

SBR

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.