DirectByteBuffer leak caused by SSL engines not closing in XNIO in the JBoss EAP 7
Environment
- Red Hat JBoss Enterprise Application Platform (EAP)
- 7.3 CP03
- 7.2 CP09
Issue
- The following exception in the JBoss server log:
ERROR [org.jboss.threads.errors] (ServerService Thread Pool -- 22) Thread Thread[ServerService Thread Pool -- 22,5,ServerService ThreadGroup] threw an uncaught exception: java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:242)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:355)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:174)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:555)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NIOPacket.readHeader(NIOPacket.java:258)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NIOPacket.readPacketFromSocketChannel(NIOPacket.java:190)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:132)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:105)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:91)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:764)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:429)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:407)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:268)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:64)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:991)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:2268)
at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.pool.OraclePooledConnection.close(OraclePooledConnection.java:242)
at org.jboss.ironjacamar.jdbcadapters@1.4.22.Final-redhat-00001//org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.destroy(XAManagedConnection.java:235)
at org.jboss.ironjacamar.impl@1.4.22.Final-redhat-00001//org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.close(XAResourceRecoveryImpl.java:412)
at org.jboss.ironjacamar.impl@1.4.22.Final-redhat-00001//org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.shutdown(XAResourceRecoveryImpl.java:157)
at org.jboss.as.connector@7.3.2.GA-redhat-00002//org.jboss.as.connector.subsystems.datasources.XaDataSourceService.stopService(XaDataSourceService.java:62)
at org.jboss.as.connector@7.3.2.GA-redhat-00002//org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$1.run(AbstractDataSourceService.java:186)
at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.base/java.lang.Thread.run(Thread.java:834)
at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Resolution
Apply Update 4 or later for EAP 7.3.
Root Cause
Diagnostic Steps
- Get a heap dump near the time of the issue and check if the majority of
DirectByteBufferinstances have a capacity of 2752512, the buffer size used by default in theSSLEnginein XNIO:
SELECT * FROM java.nio.DirectByteBuffer d WHERE (d.capacity = 2752512)
- Check that the buffers GC roots trace back to
org.xnio.ssl.JsseXnioSsl:
Class Name | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
------------------------------------------------------------------------------------------------------------------------------------------------------
[32] class org.xnio.ssl.JsseXnioSsl @ 0x266229858 | 5,670 | 16 | 362,880 | 320
'- bufferPool org.xnio.ByteBufferSlicePool @ 0x2740c3e80 | 5,670 | 48 | 362,880 | 28,744
'- directBuffers java.util.Collections$SynchronizedRandomAccessList @ 0x2740c70f0| 5,670 | 24 | 362,880 | 25,048
'- list, c java.util.ArrayList @ 0x2740c7108 | 5,670 | 24 | 362,880 | 25,024
'- elementData java.lang.Object[6246] @ 0x41c3637c8 | 5,670 | 25,000 | 362,880 | 25,000
|- [5205] java.nio.DirectByteBuffer @ 0x43092fc80 | 1 | 64 | 64 | 64
|- [5204] java.nio.DirectByteBuffer @ 0x430931ea8 | 1 | 64 | 64 | 64
|- [5599] java.nio.DirectByteBuffer @ 0x437a77498 | 1 | 64 | 64 | 64
|- [5598] java.nio.DirectByteBuffer @ 0x437a774d8 | 1 | 64 | 64 | 64
|- [5136] java.nio.DirectByteBuffer @ 0x42f70b1f0 | 1 | 64 | 64 | 64
|- [3839] java.nio.DirectByteBuffer @ 0x418dd1098 | 1 | 64 | 64 | 64
|- [547] java.nio.DirectByteBuffer @ 0x3dd27ed98 | 1 | 64 | 64 | 64
|- [3679] java.nio.DirectByteBuffer @ 0x41319be38 | 1 | 64 | 64 | 64
|- [3678] java.nio.DirectByteBuffer @ 0x41319be78 | 1 | 64 | 64 | 64
|- [3184] java.nio.DirectByteBuffer @ 0x40a595108 | 1 | 64 | 64 | 64
|- [491] java.nio.DirectByteBuffer @ 0x3ded17fd0 | 1 | 64 | 64 | 64
|- [2817] java.nio.DirectByteBuffer @ 0x402dc8cc0 | 1 | 64 | 64 | 64
|- [2816] java.nio.DirectByteBuffer @ 0x402dc8d00 | 1 | 64 | 64 | 64
|- [1864] java.nio.DirectByteBuffer @ 0x3f52759e0 | 1 | 64 | 64 | 64
|- [1863] java.nio.DirectByteBuffer @ 0x3f5275a20 | 1 | 64 | 64 | 64
|- [3951] java.nio.DirectByteBuffer @ 0x4184c5f38 | 1 | 64 | 64 | 64
|- [3950] java.nio.DirectByteBuffer @ 0x4184c5f78 | 1 | 64 | 64 | 64
|- [5287] java.nio.DirectByteBuffer @ 0x42e5db430 | 1 | 64 | 64 | 64
|- [4736] java.nio.DirectByteBuffer @ 0x4286dc638 | 1 | 64 | 64 | 64
|- [4649] java.nio.DirectByteBuffer @ 0x426f05770 | 1 | 64 | 64 | 64
|- [2994] java.nio.DirectByteBuffer @ 0x408e63a50 | 1 | 64 | 64 | 64
|- [2993] java.nio.DirectByteBuffer @ 0x408e63a90 | 1 | 64 | 64 | 64
|- [3838] java.nio.DirectByteBuffer @ 0x418dd10d8 | 1 | 64 | 64 | 64
|- [5558] java.nio.DirectByteBuffer @ 0x436656350 | 1 | 64 | 64 | 64
|- [842] java.nio.DirectByteBuffer @ 0x3e3df2658 | 1 | 64 | 64 | 64
'- Total: 25 of 5,670 entries; 5,645 more | | | |
------------------------------------------------------------------------------------------------------------------------------------------------------
Components
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.