JBoss - Java heap retention in StackTraceCapture

Environment

• JBoss Enterprise Application Platform (EAP)
  • 7.4.x
  • 8.x

Issue

• We face high heap usage on JBoss and a heap dump shows many com.arjuna.ats.arjuna.coordinator.StackTraceCapture objects are held in com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction's threadStackTraceHistoryList:

Class Name                                                                       | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
---------------------------------------------------------------------------------------------------------------------------------------------------
[15] com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction @ 0xbabb9bc0|       27,893 |          168 |           669,432 |   746,767,544
'- threadStackTraceHistoryList java.util.ArrayList @ 0xbabb9f60                  |       27,893 |           24 |           669,432 |   746,766,336
   '- elementData java.lang.Object[31618] @ 0xd44d1880                           |       27,893 |      126,488 |           669,432 |   746,766,312
      |- [2620] com.arjuna.ats.arjuna.coordinator.StackTraceCapture @ 0xcf22f380 |            1 |           24 |                24 |        26,768
      |- [15406] com.arjuna.ats.arjuna.coordinator.StackTraceCapture @ 0xb65f8490|            1 |           24 |                24 |        26,768
      |- [2619] com.arjuna.ats.arjuna.coordinator.StackTraceCapture @ 0xcf22f450 |            1 |           24 |                24 |        26,768
      |- [3204] com.arjuna.ats.arjuna.coordinator.StackTraceCapture @ 0xdb6e4ae0 |            1 |           24 |                24 |        26,768
      |- [15405] com.arjuna.ats.arjuna.coordinator.StackTraceCapture @ 0xb65fed20|            1 |           24 |                24 |        26,768
      |- [26266] com.arjuna.ats.arjuna.coordinator.StackTraceCapture @ 0xf81753e0|            1 |           24 |                24 |        26,768
---------------------------------------------------------------------------------------------------------------------------------------------------

Resolution

• Adjust the transaction timeout so an excessive value is not used and a transaction cannot remain long running and accumulate such traces longer than anticipated.
• If you do use a large transaction timeout and want to reduce the rate of trace generations during any long running transaction, then you may set the following property in your JVM to adjust that trace interval (note this trace interval is in milliseconds while the global transaction timeout setting is in seconds):

-Dcom.arjuna.ats.arjuna.coordinator.txReaperTraceInterval=1800000000

• Identify and address the cause of any excessively long running transactions.

Root Cause

• With EAP 7.4.0+, JBoss will begin to capture a stacktrace for a long running transaction every 30 seconds until it reaches the transaction timeout for diagnostic purposes
  • This content is not included.JBTM-3447
• If setting a very large transaction timeout, then this increases the total number of stacktraces that can be captured and stored for a long running transaction.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.