Java heap retention in oracle.jdbc.driver.T4CXAResource$XidListEntry

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP) 6
    • Verified in EAP 6 but may similarly impact older EAP releases
  • Oracle JDBC driver

Issue

  • Heap dump analysis shows significant memory retained by InternalManagedConnectionPool where each managed connection has an associated TxConnectionListener which retains an OracleXAResource which, in turn, owns a Hashtable of transaction identifiers where each entry contains:
    • key: org.jboss.jca.core.tx.jbossts.XidWrapperImpl
    • value: oracle.jdbc.xa.OracleXAResource$XidListEntry
Class Name                                                                                   | Retained Heap | Shallow Heap | Percentage
-----------------------------------------------------------------------------------------------------------------------------------------
|- org.jboss.jca.core.connectionmanager.listener.TxConnectionListener @ 0x769449578          |    38,931,808 |           80 |      1.75%
|  |- org.jboss.jca.adapters.jdbc.xa.XAManagedConnection @ 0x7694495c8                       |    38,931,104 |           80 |      1.75%
|  |  |- oracle.jdbc.xa.client.OracleXAResource @ 0x76941b1d0                                |    38,633,848 |           64 |      1.73%
|  |  |  |- java.util.Hashtable @ 0x768fcb888                                                |    38,633,640 |           56 |      1.73%
|  |  |  |  '- java.util.Hashtable$Entry[417791] @ 0x7ca2e5898                               |    38,633,584 |    1,671,184 |      1.73%
|  |  |  |     |- java.util.Hashtable$Entry @ 0x7b4d6ebf8                                    |       128,000 |           32 |      0.01%
|  |  |  |     |  | ...
|  |  |  |     |  |- org.jboss.jca.core.tx.jbossts.XidWrapperImpl @ 0x7dcdaeae8              |           144 |           40 |      0.00%
|  |  |  |     |  |- oracle.jdbc.xa.OracleXAResource$XidListEntry @ 0x7db342560              |            24 |           24 |      0.00%
-----------------------------------------------------------------------------------------------------------------------------------------
  • In some cases, it is observed that there are thousands of distinct entries containing unique global/branch transaction ID pairs for transactions which have completed hours or days prior to collection of the heap dump (i.e. these persist indefinitely)
    • Individually, the Xid instances retain little memory but the accumulation over time (rate of accumulation varies with application/load) becomes significant

Resolution

Upgrades to the Oracle JDBC driver may address the issue

  • Bug 5597215
    • 10.2.0.4 (Server Patch Set)
    • 11.1.0.6 (Base Release)
    • See Oracle Bug 5597215 for additional information
  • Bug 19512221
    • A patch based on Oracle JDBC Driver 11.2.0.4 is available from Oracle support for 19512221
    • Oracle JDBC Driver 12.2
    • See Oracle Doc ID 1960827.1 and Bug 19512221 for additional information
  • Contact Oracle Support for additional information regarding the above

As a workaround:

  • Set idle-timeout-minutes on your datasource so idle connections can be freed
  • Manually flush the datasource pool to free the connection objects
  • In some cases with EAP 6, adding <wrap-xa-resource>false</wrap-xa-resource> to the data source configuration will work around the issue

Root Cause

There are known bugs in certain versions of Oracle's JDBC drivers

  • Bug 5597215 : MEMORY LEAK IN XA RESOURCE AFTER APPLYING PATCH 1527725 / 4958448
    • Reproduced in the following driver versions (may not be a complete list): 10.2.0.1
  • Bug 19512221 : XARESOURCE LEAKS XID INSTANCES IF EQUIVALENT XID INSTANCES ARE USED
    • Reproduced in the following driver versions (may not be a complete list): 11.2.0.3, 11.2.0.4, 12.1.0.1

Diagnostic Steps

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.