Log4j and javax.mail are deadlocking

Solution Unverified - Updated 6 Aug 2024

Environment

JBoss Enterprise Application Platform

Issue

JBoss became unresponsive. Thread dumps show it was due to threads deadlocking between log4j SMTP appenders and direct usage of javax.mail from the application:

  "WorkManager(2)-1" J9VMThread:0x000000011877C800, j9thread_t:0x0000000118754320, java/lang/Thread:0x070000003BD50160, state:B, prio=5
  (native thread ID:0x36C0D9, native priority:0x5, native policy:UNKNOWN)
  Java callstack:
  at org/apache/log4j/Category.callAppenders(Category.java:201(Compiled Code))
  at org/apache/log4j/Category.forcedLog(Category.java:388(Compiled Code))
  at org/apache/log4j/Category.log(Category.java:849(Compiled Code))
  at org/jboss/logbridge/LogBridgeHandler.publish(LogBridgeHandler.java:72(Compiled Code))
  at java/util/logging/Logger.log(Logger.java:1121(Compiled Code))
  at java/util/logging/Logger.log(Logger.java:1021(Compiled Code))
  at com/sun/activation/registries/LogSupport.log(LogSupport.java:73)
  at javax/activation/MailcapCommandMap.createDataContentHandler(MailcapCommandMap.java:548)
  at javax/activation/DataHandler.getDataContentHandler(DataHandler.java:625)
  at javax/activation/DataHandler.writeTo(DataHandler.java:329)
  at javax/mail/internet/MimeUtility.getEncoding(MimeUtility.java:264)
  at javax/mail/internet/MimeBodyPart.updateHeaders(MimeBodyPart.java:1299)
  at javax/mail/internet/MimeMessage.updateHeaders(MimeMessage.java:2071)
  at javax/mail/internet/MimeMessage.saveChanges(MimeMessage.java:2039)
  at javax/mail/Transport.send(Transport.java:119)
  at com/package/Helper.sendDeadlockingMail(Helper.java:166)

  "WorkerThread#2[10.2.52.204:58412]" J9VMThread:0x0000000119B48400, j9thread_t:0x0000000119914A80, java/lang/Thread:0x070000003C6E2AE0, state:B, prio=5
  (native thread ID:0x2C508B, native priority:0x5, native policy:UNKNOWN)
  Java callstack:
  at javax/activation/DataHandler.getDataContentHandler(DataHandler.java:625)
  at javax/activation/DataHandler.writeTo(DataHandler.java:329)
  at javax/mail/internet/MimeUtility.getEncoding(MimeUtility.java:264)
  at javax/mail/internet/MimeBodyPart.updateHeaders(MimeBodyPart.java:1299)
  at javax/mail/internet/MimeBodyPart.updateHeaders(MimeBodyPart.java:1008)
  at javax/mail/internet/MimeMultipart.updateHeaders(MimeMultipart.java:414)
  at javax/mail/internet/MimeBodyPart.updateHeaders(MimeBodyPart.java:1285)
  at javax/mail/internet/MimeMessage.updateHeaders(MimeMessage.java:2071)
  at javax/mail/internet/MimeMessage.saveChanges(MimeMessage.java:2039)
  at javax/mail/Transport.send(Transport.java:119)
  at org/apache/log4j/net/SMTPAppender.sendBuffer(Bytecode PC:216)
  at org/apache/log4j/net/SMTPAppender.append(Bytecode PC:56)
  at org/apache/log4j/AppenderSkeleton.doAppend(AppenderSkeleton.java:230(Compiled Code))
  at org/apache/log4j/helpers/AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:65(Compiled Code))
  at org/apache/log4j/Category.callAppenders(Category.java:203(Compiled Code))
  at org/apache/log4j/Category.forcedLog(Category.java:388(Compiled Code))
  at org/apache/log4j/Category.error(Category.java:302)

Resolution

Set javax.activation to WARN level logging or higher to avoid log4j usage and potential deadlocking from direct javax.mail usage:
```
   <category name="javax.activation"> 
     <priority value="WARN"/> 
  </category>
```

Root Cause

Attempts to send emails results in java.util logging info messages, which the jboss logbridge picks up and sends through log4j. This grabs the same locks as regular log4j/SMTP appender calls, but in the inverse order, and so javax.mail calls can deadlock with the log4j SMTP appender.

Diagnostic Steps

Troubleshoot using thread dumps and additional steps mentioned in Java application unresponsive

SBR

JVM & Diagnostics

Product(s)

Red Hat JBoss Enterprise Application Platform

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.