JBoss EAP becomes unresponsive with threads blocking in org.jboss.logmanager.handlers.WriterHandler

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 6.x
  • 7.x

Issue

JBoss EAP 6.4 becomes slow/unresponsive with many threads blocking on org.jboss.logmanager.handlers.WriterHandler as in:

"http-/127.0.0.1:8080-244" daemon prio=10 tid=0x00002ac15439f800 nid=0x2e45 waiting for monitor entry [0x00002ac1750cc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.jboss.logmanager.handlers.WriterHandler.doPublish(WriterHandler.java:59)
	- waiting to lock <0x000000040002d8b0> (a java.lang.Object)
	at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:79)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:292)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:300)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:300)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:300)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:300)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:300)
	at org.jboss.logmanager.Logger.logRaw(Logger.java:721)
	at org.apache.log4j.Category.callAppenders(Category.java:66)
	at org.apache.log4j.Category.forcedLog(Category.java:118)
	at org.apache.log4j.Category.info(Category.java:226)

Resolution

These are standard steps that can decrease the impact of logging overhead:

• Decrease logging verbosity.
• Fix any potential issues seen that can cause floods of frequent logging messages
• Disable the console appender
• Switch to the Async Log Handler

Root Cause

Threads are backed up in logging, i.e. to Publish.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.