Java heap and direct buffers are consumed by thousands of remoting connections created by client EJB calls

Environment

• JBoss Enterprise Application Platform (EAP) 7.x

Issue

• During load in our EJB client app, our heap is consumed by thousands of remote connection objects, on the client and server side. Socket counts grow with this on the client and server side and this eventually leads to a java.lang.OutOfMemoryError: Direct buffer memory failure.

Resolution

• Identify and remove any recurring SSLContext.setDefault() calls in the client application code.

Related Articles

• How configure an EJB client in EAP 7.1+

Root Cause

• The client application is making recurring SSLContext.setDefault() calls with new SSLContexts.
• It's expected for a remoting client to open and reuse a single connection. That connection is keyed by an SSLContext. The SSLContext used to fetch the connection for reuse is pulled Content from github.com is not included.here from the remoting Endpoint, which can then ultimately grab it from org.wildfly.security.auth.client.AuthenticationContextConfigurationClient.getSSLContextFactory, which we can see can pull from the JVM SSLContext default:

  public SecurityFactory<SSLContext> getSSLContextFactory(URI uri, AuthenticationContext authenticationContext, String abstractType, String abstractTypeAuthority) {
    Assert.checkNotNullParam("uri", uri);
    Assert.checkNotNullParam("authenticationContext", authenticationContext);
    RuleNode<SecurityFactory<SSLContext>> node = authenticationContext.sslRuleMatching(uri, abstractType, abstractTypeAuthority);
    if (node == null) return SSLContext::getDefault; 
    return (SecurityFactory)node.getConfiguration();
  }

• Thus any SSLContext.setDefault call in the client application code can break remoting connection reuse to the point that connections grow continually until failing due to some OOME or file limit failure.

Diagnostic Steps

• Capture output to check socket counts and their state:

netstat -vatnp

• Capture a heap dump from the OOME state. With this heap dump, use the following OQL to confirm what ip/port the remoting connections are accumulating for:

SELECT connection.sslConduitEngine.sinkConduit.socketChannel.remoteAddress.holder.addr.holder.address, connection.sslConduitEngine.sinkConduit.socketChannel.remoteAddress.holder.port, connection.sslConduitEngine.sinkConduit.socketChannel.localAddress.holder.addr.holder.address, connection.sslConduitEngine.sinkConduit.socketChannel.localAddress.holder.port FROM org.jboss.remoting3.remote.RemoteConnection where sslChannel != null

• Use the following OQL to determine if there are unique connection keys being created with varied SSLContexts to prevent proper connection reuse:

SELECT hashCode, realUri.string.value.toString(), realUri.hash, sslContext FROM org.jboss.remoting3.ConnectionKey 

• Use byteman during a reproducer to check for the source of unexpected new SSLContexts used by the EJB Client. Create a text file named RemotingConnection.btm with rule content like below:

RULE doGetConnection
CLASS ^org.jboss.remoting3.EndpointImpl
METHOD doGetConnection
AT EXIT
IF TRUE
DO traceStack("--------------------------->EndpointImpl.doGetConnection " + $1 + " " + $2 + " " + $! + "\n");
ENDRULE

RULE SSLContext
CLASS ^javax.net.ssl.SSLContext
METHOD <init>
AT ENTRY
IF TRUE
DO traceStack("--------------------------->SSLContext.<init>\n");
ENDRULE

• Enable the byteman agent with a JVM option like below, and restart, and reproduce:

-javaagent:<mypath>/lib/byteman.jar=script:<mypath>/RemotingConnection.btm,boot:<mypath>/lib/byteman.jar -Dorg.jboss.byteman.transform.all

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.