JBoss Controller reaches 'OutOfMemoryError: Direct buffer memory' after many connection retries

Environment

• Red Hat JBoss Enterprise Application Platform (EAP) 7.x running in domain mode

Issue

• In our domain controller, we are seeing XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory getting logged continuously every second.

Resolution

• Address any issues causing the master to be unresponsive or unreachable. Restart slave controllers after that is corrected to clean up any accumulated retry connections if that state persisted for a long time.
• Update to EAP 7.3.4+

Root Cause

• This content is not included.JBEAP-20128
• This content is not included.JBEAP-20130
• This content is not included.WFCORE-5105
• This content is not included.XNIO-380
• The domain master was unresponsive or unreachable previously due to some other event (such as a heap space OutOfMemoryError). This causes slaves to retry their connection to the master frequently:

INFO  [org.jboss.as.host.controller] (Host Controller Service Threads - 464) WFLYHC0150: Trying to reconnect to master host controller.

• The slave controller leak these connections with each retry. So if the master issue state persists for a long period of time (e.g. multiple days), this can cause connections to accumulate to a great number over time with each slave connection re-attempt to the master. The accumulation of these slave connections can eventually swamp controllers, causing the "OutOfMemoryError: Direct buffer memory".

Diagnostic Steps

• Capture a heap dump from both the master controller process and a slave when OutOfMemoryError happens.
• Check the the number of org.xnio.nio.NioSocketConduit connections in the master and slave heap dumps:

SELECT socketChannel.localAddress.holder.addr.holder.address, socketChannel.localAddress.holder.port, socketChannel.remoteAddress.holder.addr.holder.address, socketChannel.remoteAddress.holder.port FROM org.xnio.nio.NioSocketConduit 

• Check if the connections are to the management port:

        <management-interfaces>
           <native-interface security-realm="ManagementRealm">
                <socket interface="management" port="9990"></socket>
            </native-interface>

• Collect netstat -vatnp output from the master and slave.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.