JBoss deadlocks in LogManager

Solution Verified - Updated 6 Aug 2024

Environment

JBoss Enterprise Application Platform (EAP) 6
- 6.0.1

Issue

JBoss becomes unresponsive and the thread dumps show threads heavily stalled in logging due to a deadlock in log manager code:

"http-/0.0.0.0:8080-815":
	at java.io.PrintStream.println(PrintStream.java:805)
	- waiting to lock <0x9ffffffe24c7dd58> (a java.io.PrintStream)
	at org.jboss.stdio.StdioContext$DelegatingPrintStream.println(StdioContext.java:297)
	at java.util.logging.ErrorManager.error(ErrorManager.java:94)
	- locked <0x9ffffffe218b3810> (a java.util.logging.ErrorManager)
	at java.util.logging.Handler.reportError(Handler.java:232)
	at org.jboss.logmanager.handlers.WriterHandler.safeFlush(WriterHandler.java:175)
	at org.jboss.logmanager.handlers.WriterHandler.flush(WriterHandler.java:142)
	- locked <0x9ffffffe218b3840> (a java.lang.Object)
	at org.jboss.logmanager.ExtHandler.doPublish(ExtHandler.java:104)
	at org.jboss.logmanager.handlers.WriterHandler.doPublish(WriterHandler.java:70)
	- locked <0x9ffffffe218b3840> (a java.lang.Object)
	at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:76)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:290)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.Logger.logRaw(Logger.java:721)
	at org.jboss.logmanager.Logger.logRaw(Logger.java:731)
	at org.jboss.logmanager.Logger.log(Logger.java:367)
	...

"http-/0.0.0.0:8080-1409":
	at org.jboss.logmanager.handlers.WriterHandler.doPublish(WriterHandler.java:59)
	- waiting to lock <0x9ffffffe218b3840> (a java.lang.Object)
	at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:76)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:290)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:298)
	at org.jboss.logmanager.Logger.logRaw(Logger.java:721)
	at org.jboss.logmanager.Logger.log(Logger.java:506)
	at org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71)
	- locked <0x9ffffffe24c7de00> (a java.lang.StringBuilder)
	at org.jboss.stdio.WriterOutputStream.finish(WriterOutputStream.java:143)
	at org.jboss.stdio.WriterOutputStream.flush(WriterOutputStream.java:164)
	- locked <0x9ffffffe24c7df18> (a sun.nio.cs.UTF_8$Decoder)
	at java.io.PrintStream.write(PrintStream.java:482)
	- locked <0x9ffffffe24c7dd58> (a java.io.PrintStream)
	at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
	at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
	at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
	- locked <0x9ffffffe24c7e340> (a java.io.OutputStreamWriter)
	at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
	at java.io.PrintStream.newLine(PrintStream.java:546)
	- locked <0x9ffffffe24c7dd58> (a java.io.PrintStream)
	at java.io.PrintStream.println(PrintStream.java:807)
	- locked <0x9ffffffe24c7dd58> (a java.io.PrintStream)
	at org.jboss.stdio.StdioContext$DelegatingPrintStream.println(StdioContext.java:297)
	at java.util.logging.ErrorManager.error(ErrorManager.java:94)
	- locked <0x9ffffffe212bf3c0> (a java.util.logging.ErrorManager)
	at java.util.logging.Handler.reportError(Handler.java:232)
	at org.jboss.logmanager.handlers.WriterHandler.doPublish(WriterHandler.java:73)
	at org.jboss.logmanager.ExtHandler.publish(ExtHandler.java:76)
	at org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:290)
	at org.jboss.logmanager.Logger.logRaw(Logger.java:721)
	at org.jboss.logmanager.Logger.fine(Logger.java:481)
	...

Resolution

Await a fix to this bug in a later release

Root Cause

Content from issues.jboss.org is not included.LOGMGR-65
Two threads were handling log level errors, which were being sent through stderr. JBoss was sending stderr through these same loggers and so the two threads can deadlock across the PrintStreams and WriterHandlers used.

Diagnostic Steps

Troubleshoot using thread dumps and additional steps mentioned in Java application unresponsive

SBR

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.