OutOfMemory GC Overhead using Oracle JDBC Driver

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)
  • Oracle JDBC

Issue

  • We are facing issues in our production environment with the server hitting very high peaks in memory and going out of memory. (GC overhead)
  • A heap dump shows most of the memory taken up in oracle.jdbc.driver.BufferCaches referenced from oracle T4CConnections' oracle.jdbc.driver.PhysicalConnection$BufferCacheStores.
  • A heap dump shows the following retention under a single instance of org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool:
Class Name                                                                                 | Shallow Heap | Retained Heap | Percentage
--------------------------------------------------------------------------------------------------------------------------------------
org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool @ ...|           72 |   730,013,848 |     66.37%
|- java.util.ArrayList @ 0xa80022f0                                                        |           24 |   730,008,864 |     66.37%
|  '- java.lang.Object[300] @ 0xa8002308                                                   |        1,216 |   730,008,840 |     66.37%
|     |- org.jboss.jca.core.connectionmanager.listener.NoTxConnectionListener @ ...        |           72 |    10,806,344 |      0.98%
|     |  |- org.jboss.jca.adapters.jdbc.local.LocalManagedConnection @ 0xaf882c78          |           72 |    10,806,168 |      0.98%
|     |  |  |- oracle.jdbc.driver.T4CConnection @ 0xaf882cc0                               |          840 |    10,805,152 |      0.98%
|     |  |  |  |- oracle.jdbc.driver.PhysicalConnection$BufferCacheStore @ 0xaf885618      |           24 |    10,675,088 |      0.97%
|     |  |  |  |  |- oracle.jdbc.driver.BufferCache @ 0xaf885d10                           |           32 |    10,578,624 |      0.96%
...
--------------------------------------------------------------------------------------------------------------------------------------

Resolution

  • Set -Doracle.jdbc.maxCachedBufferSize=18 as a Java System Property to limit buffer cache sizes
  • The size of the BufferedCache in the T2Connection are Oracle constructs. You should contact Oracle for further guidance.

Root Cause

  • Oracle connection BufferedCache's are unbounded by default. They are referenced With SoftReferences though and so can always be GC'd whenever needed. If they grow too large and/or the collector is too slow to free them, the caches could still lead to GC Overhead OOMEs.

Diagnostic Steps

  • Get a heap dump at the time of the problem

It should show there are 15 Oracle JDBC Connections which are using a using the majority of the heap space. Individually they range from 85mb to 210mb.

The memory is primarily being used for the driver buffer cache. They are held in soft references so it wouldn't actually run out of memory, but it could still cause high GC activity and an OutOfMemoryError due to overhead.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.