JBoss - Java heap retention in BasicThreadPool$TimeoutInfo

Environment

• JBoss Enterprise Application Platform

Issue

• "java.lang.OutOfMemoryError: Java heap space" errors are received  because the heap is consumed by org.jboss.util.threadpool.BasicThreadPool$TimeoutInfo

• This BasicThreadPool$TimeoutInfo is stored in a single instance of EDU.oswego.cs.dl.util.concurrent.Heap called tasksWithTimeouts by the TimeoutMonitor thread (another subclass of BasicThreadPool).

Resolution

• Ensure the TimeoutMonitor thread is alive and running well as it times out tasks and iterates through and removes the TimeoutInfo.

Root Cause

• The TimeoutMonitor thread's responsibility is to iterate through and remove the TimeoutInfo in tasksWithTimeouts.  It attempts to stop any tasks as needed that have exceeded their timeout without completing.
• If something goes wrong with the TimeoutMonitor thread, these TimeoutInfo objects can build up.

Diagnostic Steps

• Troubleshoot with a heap dump as described in Java application "java.lang.OutOfMemoryError: Java heap space" to identify the source of retention.

• Enable the trace logging for org.jboss.util.threadpool.BasicThreadPool to see historical data as the TimeoutMonitor thread progresses.  You'd likely want to send this to its own separate log file appender.  Ensure this appender uses the %t option so that log messages can be correlated with threads.

• Gather a series of thread dumps close to the time of the OutOfMemoryError (OOME) to check the status of the TimeoutMonitor thread.

  • How do I generate a Java thread dump on Linux/Unix?
  • How do I generate a Java thread dump on Windows?

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.