Large heap retention in arjuna TransactionImple in JBoss EAP

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 6.2
    • 6.3
    • 6.4

Issue

  • We hit an OOME and the heap dump shows hundreds of thousnands of instances of com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple in the heap.
    • These objects are stored in the TransactionImple class's static _transactions map.
  • Possible memory leak in TransactionImple._transactions

Resolution

This issue is resolved in This content is not included.EAP 6.4.0 cumulative patch 2 (6.4.2) or later

Root Cause

Diagnostic Steps

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Example Eclipse Memory Analysis Tool (MAT) dominator_tree report

Class Name                                                                                       ... Retained Heap | Percentage
-------------------------------------------------------------------------------------------------------------------------------
class com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple @ 0xfffffffe20f59088   ... 4,020,202,544 |     96.75%
|- java.util.concurrent.ConcurrentHashMap @ 0xfffffffe20f59120                                   ... 4,020,202,496 |     96.75%
|  |- java.util.concurrent.ConcurrentHashMap$Node[4096] @ 0xffffffff15f86398                     ... 4,020,202,360 |     96.75%
|  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0xfffffffe861d5b80                        ...    13,391,456 |      0.32%
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0xfffffffe861d5ba0                     ...     9,006,112 |      0.22%
|  |  |  |- com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple @ 0xfffffffe62ef6ac0 ... 4,385,096 |      0.11%
|  |  |  |- com.arjuna.ats.arjuna.common.Uid @ 0xfffffffe62ef6a60                                ...           216 |      0.00%
|  |  ...
-------------------------------------------------------------------------------------------------------------------------------

Use the following OQL to check the heap for a high volume of ABORTED transactions

select * from com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple where _theTransaction.actionStatus = 4

Inspect the server log for events such as the following

... WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX <timed-out-transaction-id-here> in state  RUN

Each status number might indicate one different status. Check more about it in Content from github.com is not included.source code

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.