High CPU due to RMI TCP Connection thread infinite loop on HashMap

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)

Issue

  • JBoss is consuming high CPU and it appears to be from many RMI TCP Connection threads looping on org.jboss.resource.adapter.jdbc.remote.WrapperDataSourceService's 'connectionMap' HashMap:
    "RMI TCP Connection(14353)-119.43.98.45" daemon prio=6 tid=0x4e283c00 nid=0x22a4 runnable [0x54c4f000]
   java.lang.Thread.State: RUNNABLE
     at java.util.HashMap.put(HashMap.java:374)
     at org.jboss.resource.adapter.jdbc.remote.WrapperDataSourceService.createConnectionProxy(WrapperDataSourceService.java:499)
     at org.jboss.resource.adapter.jdbc.remote.WrapperDataSourceService.doDataSourceMethod(WrapperDataSourceService.java:347)
     at org.jboss.resource.adapter.jdbc.remote.WrapperDataSourceService.invoke(WrapperDataSourceService.java:201)
     at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
     at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
     at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
     at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
     at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
     at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:818)
     at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:419)
     at sun.reflect.GeneratedMethodAccessor133.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
     at sun.rmi.transport.Transport$1.run(Transport.java:159)
     at java.security.AccessController.doPrivileged(Native Method)
     at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
     at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
     at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
     at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     at java.lang.Thread.run(Thread.java:619)

Resolution

  • This HashMap usage ultimately stems from remote datasource access.  Remote  access is not recommended due to many performance concerns.  The best  option would be to remove
    <use-java-context>false</use-java-context>
    from all datasources and access them through the "java:/" context.
* Use the following [one-off](https://access.redhat.com/jbossnetwork/restricted/softwareDetail.html?softwareId=11113&product=appplatform&version=4.3.0.GA_CP08&downloadType=patches) for EAP 4.3.0_CP08 * Use the following [one-off](https://access.redhat.com/jbossnetwork/restricted/softwareDetail.html?softwareId=9563) for EAP 5.1.0 * Upgrade to EAP 5.1.2 or later to acquire the fix.

Root Cause

Diagnostic Steps

  • Troubleshoot with thread dumps and CPU utilization data as described in Java application high CPU to identify looping threads and high CPU consumers
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.