JBoss stalls with many threads blocking in AsyncHandler

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)
  • 6

Issue

  • Our JBoss server becomes unresponsive and thread dumps indicate that many threads were stalled blocking like so on the AsyncHandler:
"http-127.0.0.1:8080-1" daemon prio=10 tid=0x00007f0c14036800 nid=0x3cda waiting on condition [0x00007f0be2be8000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000007032c4458> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:324)
	at org.jboss.logmanager.handlers.AsyncHandler.doPublish(AsyncHandler.java:148)
	at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:76)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:290)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.Logger.logRaw(Logger.java:721)
	at org.jboss.logmanager.Logger.log(Logger.java:672)
	at org.jboss.logmanager.Logger.log(Logger.java:684)
	at org.jboss.logging.JBossLogManagerLogger.doLog(JBossLogManagerLogger.java:44)

Resolution

  • Decrease logging verbosity so the queue is less likely to fill
  • Avoid heavy weight appenders that would cause slow queue emptying
  • Avoid using appenders that will generate logging events themselves. Don't call System.err/System.out in log appenders.
  • Increase the queue length of the async handler so that it is less likely to fill:
        <async-handler name="ASYNC">
           <queue-length value="1024"/>
  • Set the async handler's overflow action to discard. Threads will no longer block in the event of a full queue, but the downside is that the logging message will be discarded:
        <async-handler name="ASYNC">
           <overflow-action value="discard"/>

Root Cause

  • With an AsyncHandler, threads write logging events to a queue in memory and a background AsyncHandler$AsyncTask thread writes out the queue contents. If the queue is full, threads will block by default waiting for room to place their logging entries.
  • A stalled AsyncTask logger thread could cause long thread waits on a full queue.
  • If the AsyncTask calls any appenders that try to log to this same logging queue, a dead lock can occur if it fills.

Diagnostic Steps

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.