JBoss - Java heap retention in PeriodicRecovery thread's '_xidScans' HashTable

Environment

• JBoss Enterprise Application Platform (EAP)
  • 5.1.x

Issue

• We're receiving OutOfMemoryErrors, and the heap dump shows the heap being consumed by the PeriodicRecovery thread's HashTable called '_xidScans'

• The server.log shows DEBUG messages like the following:

        DEBUG [org.jboss.resource.connectionmanager.ManagedConnectionFactoryDeployment] (Thread-19) Unable to execute isValid() due to null
  

Resolution

• Use a JDBC 4 compliant database driver (compiled with JDK 1.6) that provides the java.sql.Connection.isValid(int) method
  • For db2 drivers, they would need to be version 9.5 or later. See http://publib.boulder.ibm.com/infocenter/db2luw/v9r5/index.jsp?topic=%2Fcom.ibm.db2.luw.wn.doc%2Fdoc%2Fc0052012.html
• Acquire the fix to JBTM-924 to address relevant recovery leaks. Upgrade to EAP 5.2.0+ or the following one-offs are available:
  • This content is not included.5.1.1
  • This content is not included.5.1.2

Root Cause

• Database driver does not provide the isValid(int) method
• This leads to validation failure and managed connections being recreated without old bad ones being removed, causing unwanted growth in '_xidScans'
• Content from issues.jboss.org is not included.JBTM-924

Diagnostic Steps

• Troubleshoot with a heap dump as described in Java application "java.lang.OutOfMemoryError: Java heap space" to identify the source of retention.
• Check for abnormal growth in a HashTable/array called '_xidScans' belonging to the PeriodicRecovery thread.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.