JBoss web requests become slow when using a queueless executor pool

Environment

• JBoss Enterprise Application Platform (EAP) 6.x

Issue

• We've configured our web connector to use a queueless executor pool. We notice over time that response times start to degrade.
• The access log doesn't indicate an increase in response time so the delay is seen between when JBoss accepts the connection and when the access log is invoked.
• Thread dumps seem to show just idle executor threads.

Resolution

• Do not use a queueless executor. Use the default connector pool or a queued executor instead.
• Upgrade to EAP 6.4.0 or later

Root Cause

• Content from issues.jboss.org is not included.JBTHR-27
• This content is not included.BZ-1089075
• There is a race condition in the QueuelessExecutor that is allowed by the interplay between the waitingWorker being set to null both in the worker run's finally block and the executor's execute call. If the timing is just right, this can allow multiple threads to become the waitingWorker and wait on the taskEnqueued condition (only one thread should ever wait on taskEnqueued). As more threads start to slip through and wait on taskEnqueued, a delay increases between the time that a waitingWorker is assigned a task and the time that it is actually signaled so that it can start executing that task.

Diagnostic Steps

• Capture thread dumps. You will likely see idle executor threads mostly, but you can determine if this issue has occurred by checking line numbers on the waiting executors.
• This thread is an expected idle executor thread for EAP 6.2.0 at line QueuelessExecutor.java:641:

"HTTP-128" prio=10 tid=0x000000001b44c000 nid=0x61c9 waiting on condition [0x000000005565b000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074ea2a2f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176)
	at org.jboss.threads.QueuelessExecutor$Worker.awaitTimed(QueuelessExecutor.java:597)
	at org.jboss.threads.QueuelessExecutor$Worker.run(QueuelessExecutor.java:641)
	at java.lang.Thread.run(Thread.java:744)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

• This thread is a waitingWorker waiting on taskEnqueued. There should only be one. If there are multiple threads on QueuelessExecutor.java:651 for EAP 6.2.0, then you could be seeing delays from this issue:

"HTTP-139" prio=10 tid=0x000000001b990000 nid=0x25f0 waiting on condition [0x0000000055f64000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074ea2a2b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176)
	at org.jboss.threads.QueuelessExecutor$Worker.awaitTimed(QueuelessExecutor.java:597)
	at org.jboss.threads.QueuelessExecutor$Worker.run(QueuelessExecutor.java:651)
	at java.lang.Thread.run(Thread.java:744)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.