JBoss - Java heap retention in T4CPreparedStatements

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform
  • Oracle

Issue

  • This content is not included."java.lang.OutOfMemoryError: Java heap space" errors are received because the heap is consumed by oracle.jdbc.driver.T4CPreparedStatement.
  • The T4CPreparedStatements are abnormally large (several megabytes each) because of a large char[] they own called defineChars.  However, this defineChars char[] appears to only contain white space.

Resolution

  • Increase heap size to allow room for the cached statements.

  • Lower the <prepared-statement-cache-size> as needed to avoid the OOME.

  • The defineChars char[]s is buffer space preallocated by the Oracle db driver sized ahead of time for the largest possible column value for a result set. You may need to rethink the queries/columns that are causing such large buffer spaces to be created for these prepared statements.

  • As a workaround, try flushing the datasource connection pool when heap usage is growing high to free the connections and their cached prepared statements:

    1. Open the JMX Console (e.g. http://localhost:8080/jmx-console/).

    2. Scroll down to the "jboss.jca" section.

    3. Click the  ManagedConnectionPool link for the appropriate datasource.  The link will be  "name=<dsName>,service=ManagedConnectionPool" where <dsName>  is the name of the datasource.

    4. Scroll down and invoke the "flush" operation.

  • If seeing this growth under T4CConnections from possibly unclosed statements, then enable statement tracking on your datasource with the following setting to ensure those are cleaned up:

                        <track-statements>true</track-statements>

If a PreparedStatement that has not been closed is detected, the below log IJ030040 will be logged when java.sql.Connection.close() is called:

11:14:30,425 WARN  [org.jboss.jca.adapters.jdbc.WrappedConnection] (default task-1) IJ030040: Closing a statement you left open, please do your own housekeeping for: java:jboss/datasources/OracleDS: java.lang.Throwable: STACKTRACE
	at org.jboss.jca.adapters.jdbc.WrappedConnection.registerStatement(WrappedConnection.java:2075)
	at org.jboss.jca.adapters.jdbc.WrappedStatement.<init>(WrappedStatement.java:129)
	at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.<init>(WrappedPreparedStatement.java:84)
	at org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.<init>(WrappedPreparedStatement.java:69)
	at org.jboss.jca.adapters.jdbc.jdk8.WrappedPreparedStatementJDK8.<init>(WrappedPreparedStatementJDK8.java:53)
	at org.jboss.jca.adapters.jdbc.jdk8.WrappedConnectionJDK8.wrapPreparedStatement(WrappedConnectionJDK8.java:83)
	at org.jboss.jca.adapters.jdbc.WrappedConnection.prepareStatement(WrappedConnection.java:482)
	at org.example.servlet.ConnectionValidator.doGet(ConnectionValidator.java:33)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:503)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
        ...

Root Cause

  • These are cached PreparedSatements stored in the 'psCache' belonging to connections from the datasource pool, which is enabled by the following setting in a *-ds.xml:
    <prepared-statement-cache-size>150</prepared-statement-cache-size>
  • Every connection of a datasource owns a psCache with a size specified by the above parameter. The psCache is not cleared when its owner connection is returned to the pool. If the application gets the connection again, it will reuse prepared statements from this cache.
  • If the connection manager destroys a connection, its psCache will be removed as well. The psCache is a property of a managed connection; if the connection is destroyed, so will the psCache be destroyed.
  • There is not enough heap to support all the cached prepared statements. There could be a maximum of <prepared-statement-cache-size> * <max-pool-size> cached prepared statements in the heap.
  • The defineChars char[]s are very large, causing the prepared statements to consume lots of heap when stored in the cache. 
* If the T4CPreparedStatements are referenced from a T4CConnection, then this may be the result of unclosed statements instead of statement caching.

Diagnostic Steps

Useful OQL for Investigating through Eclipse MAT

  • Select for T4CPreparedStatements:

    SELECT * FROM oracle.jdbc.driver.T4CPreparedStatement

  • Select for T4CPreparedStatements over a specified size(in bytes):

    SELECT * FROM oracle.jdbc.driver.T4CPreparedStatement s WHERE s.@retainedHeapSize > ###
  • SELECT to check the values of defineChars char[]s for prepared statements within a specified size:
    
SELECT toString(s.defineChars) FROM oracle.jdbc.driver.T4CPreparedStatement s WHERE ((s.@retainedHeapSize>###)and(s.@retainedHeapSize<###))

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.