DirectByteBuffer leak caused by SSL engines not closing in XNIO in the JBoss EAP 7

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 7.3 CP03
    • 7.2 CP09

Issue

  • The following exception in the JBoss server log:
ERROR [org.jboss.threads.errors] (ServerService Thread Pool -- 22) Thread Thread[ServerService Thread Pool -- 22,5,ServerService ThreadGroup] threw an uncaught exception: java.lang.OutOfMemoryError: Direct buffer memory
        at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
        at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
        at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317)
        at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242)
        at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:242)
        at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
        at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:355)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:174)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:555)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NIOPacket.readHeader(NIOPacket.java:258)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NIOPacket.readPacketFromSocketChannel(NIOPacket.java:190)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:132)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:105)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:91)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:764)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:429)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:407)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:268)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:64)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:991)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:2268)
        at com.oracle.ojdbc8@19.3.0.0.0//oracle.jdbc.pool.OraclePooledConnection.close(OraclePooledConnection.java:242)
        at org.jboss.ironjacamar.jdbcadapters@1.4.22.Final-redhat-00001//org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.destroy(XAManagedConnection.java:235)
        at org.jboss.ironjacamar.impl@1.4.22.Final-redhat-00001//org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.close(XAResourceRecoveryImpl.java:412)
        at org.jboss.ironjacamar.impl@1.4.22.Final-redhat-00001//org.jboss.jca.core.tx.jbossts.XAResourceRecoveryImpl.shutdown(XAResourceRecoveryImpl.java:157)
        at org.jboss.as.connector@7.3.2.GA-redhat-00002//org.jboss.as.connector.subsystems.datasources.XaDataSourceService.stopService(XaDataSourceService.java:62)
        at org.jboss.as.connector@7.3.2.GA-redhat-00002//org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$1.run(AbstractDataSourceService.java:186)
        at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
        at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
        at java.base/java.lang.Thread.run(Thread.java:834)
        at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.JBossThread.run(JBossThread.java:485)

Resolution

Apply Update 4 or later for EAP 7.3.

Root Cause

This content is not included.JBEAP-20243 - XNIO-380 - SSL connection with START_TLS where TLS is not started leaks direct buffers

Diagnostic Steps

  • Get a heap dump near the time of the issue and check if the majority of DirectByteBuffer instances have a capacity of 2752512, the buffer size used by default in the SSLEngine in XNIO:
SELECT * FROM java.nio.DirectByteBuffer d WHERE (d.capacity = 2752512)
  • Check that the buffers GC roots trace back to org.xnio.ssl.JsseXnioSsl:
Class Name                                                                          | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
------------------------------------------------------------------------------------------------------------------------------------------------------
[32] class org.xnio.ssl.JsseXnioSsl @ 0x266229858                                   |        5,670 |           16 |           362,880 |           320
'- bufferPool org.xnio.ByteBufferSlicePool @ 0x2740c3e80                            |        5,670 |           48 |           362,880 |        28,744
   '- directBuffers java.util.Collections$SynchronizedRandomAccessList @ 0x2740c70f0|        5,670 |           24 |           362,880 |        25,048
      '- list, c java.util.ArrayList @ 0x2740c7108                                  |        5,670 |           24 |           362,880 |        25,024
         '- elementData java.lang.Object[6246] @ 0x41c3637c8                        |        5,670 |       25,000 |           362,880 |        25,000
            |- [5205] java.nio.DirectByteBuffer @ 0x43092fc80                       |            1 |           64 |                64 |            64
            |- [5204] java.nio.DirectByteBuffer @ 0x430931ea8                       |            1 |           64 |                64 |            64
            |- [5599] java.nio.DirectByteBuffer @ 0x437a77498                       |            1 |           64 |                64 |            64
            |- [5598] java.nio.DirectByteBuffer @ 0x437a774d8                       |            1 |           64 |                64 |            64
            |- [5136] java.nio.DirectByteBuffer @ 0x42f70b1f0                       |            1 |           64 |                64 |            64
            |- [3839] java.nio.DirectByteBuffer @ 0x418dd1098                       |            1 |           64 |                64 |            64
            |- [547] java.nio.DirectByteBuffer @ 0x3dd27ed98                        |            1 |           64 |                64 |            64
            |- [3679] java.nio.DirectByteBuffer @ 0x41319be38                       |            1 |           64 |                64 |            64
            |- [3678] java.nio.DirectByteBuffer @ 0x41319be78                       |            1 |           64 |                64 |            64
            |- [3184] java.nio.DirectByteBuffer @ 0x40a595108                       |            1 |           64 |                64 |            64
            |- [491] java.nio.DirectByteBuffer @ 0x3ded17fd0                        |            1 |           64 |                64 |            64
            |- [2817] java.nio.DirectByteBuffer @ 0x402dc8cc0                       |            1 |           64 |                64 |            64
            |- [2816] java.nio.DirectByteBuffer @ 0x402dc8d00                       |            1 |           64 |                64 |            64
            |- [1864] java.nio.DirectByteBuffer @ 0x3f52759e0                       |            1 |           64 |                64 |            64
            |- [1863] java.nio.DirectByteBuffer @ 0x3f5275a20                       |            1 |           64 |                64 |            64
            |- [3951] java.nio.DirectByteBuffer @ 0x4184c5f38                       |            1 |           64 |                64 |            64
            |- [3950] java.nio.DirectByteBuffer @ 0x4184c5f78                       |            1 |           64 |                64 |            64
            |- [5287] java.nio.DirectByteBuffer @ 0x42e5db430                       |            1 |           64 |                64 |            64
            |- [4736] java.nio.DirectByteBuffer @ 0x4286dc638                       |            1 |           64 |                64 |            64
            |- [4649] java.nio.DirectByteBuffer @ 0x426f05770                       |            1 |           64 |                64 |            64
            |- [2994] java.nio.DirectByteBuffer @ 0x408e63a50                       |            1 |           64 |                64 |            64
            |- [2993] java.nio.DirectByteBuffer @ 0x408e63a90                       |            1 |           64 |                64 |            64
            |- [3838] java.nio.DirectByteBuffer @ 0x418dd10d8                       |            1 |           64 |                64 |            64
            |- [5558] java.nio.DirectByteBuffer @ 0x436656350                       |            1 |           64 |                64 |            64
            |- [842] java.nio.DirectByteBuffer @ 0x3e3df2658                        |            1 |           64 |                64 |            64
            '- Total: 25 of 5,670 entries; 5,645 more                               |              |              |                   |              
------------------------------------------------------------------------------------------------------------------------------------------------------
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.