BaseClassLoader's blackList causing memory leaks in JBoss

Environment

• JBoss Enterprise Application Platform (EAP) 5
• JBossWS
• Groovy

Issue

• We have a memory leak using JBossWS. The heap dump shows the retention is in BaseClassLoader's and their blackList HashMaps.
• Our application uses groovy and hits an OOME. The heap dump shows the retention is in BaseClassLoader's and their blackList HashMaps. The blacklist holds many similar entries for SimpleTemplateScript####BeanInfo

Resolution

• If using JBossWS and seeing blacklist growth from its temporary file references, upgrade to EAP 5.2.0 where that temporary file behavior and the resulting blacklisting should be changed and avoided.
• As a workaround, the blackList can be manually flushed at any time. There are several ways to do this:

1. Find the jboss.classloader in the jmx-console. You will see many links here for mbeans to instances of org.jboss.classloader.spi.base.BaseClassLoader (named id="..."). At the bottom of these pages, you will see a clearBlackList operation (you'd want the one with no parameters), which you can invoke to empty the blackList cache.

2. This same operation can be invoked through twiddle, for example:

    twiddle.sh -u admin -p admin -s jnp://localhost:1099 invoke jboss.classloader:'id="jmx-classloader:0.0.0$MODULE"' clearBlackList
   

   And you can check the blackList cache size as well:

    twiddle.sh -u admin -p admin -s jnp://localhost:1099 get jboss.classloader:'id="jmx-classloader:0.0.0$MODULE"' ResourceBlackListSize
   

3. Call the clearBlackList method through the mbean programmatically:

    import javax.management.MBeanServer;
    import javax.management.MalformedObjectNameException;
    import org.jboss.mx.util.MBeanProxyExt;
    import org.jboss.mx.util.MBeanServerLocator;
    import org.jboss.classloader.spi.base.BaseClassLoaderMBean;
    
    public class Sample {
      public void work() {
        MBeanServer server = MBeanServerLocator.locateJBoss();
        BaseClassLoaderMBean mbean = null;
    
        try {
          mbean = (BaseClassLoaderMBean) MBeanProxyExt.create(BaseClassLoaderMBean.class, "jboss.classloader:id=\"jmx-classloader:0.0.0$MODULE\"", server);
        } catch (MalformedObjectNameException e) {
          e.printStackTrace();
        }
    
        mbean.clearBlackList();
      }
    }
   

• For another workaround, try disabling the classloaders blacklisting by adding a WEB-INF/jboss-classloading.xml that contains:

<classloading xmlns="urn:jboss:classloading:1.0"
	      domain="IsolatedDomain"
	      export-all="NON_EMPTY"
	      import-all="true"
	      blackList="false">
</classloading>

Root Cause

• The growth is in the BaseClassLoader's blackList HashMap. Entries are added to this hashmap whenever a resource is looked up and not found. That way the classloader can check the blackList and quickly return a null for any repeat lookups for non-existant resources
• There are no bounds on this cache currently in EAP 5.
• This can lead to large, unwanted growth over time if something looks up and misses a new resource frequently (such as a dynamically fetched web service schema file).
  • Content from issues.jboss.org is not included.JBPAPP-9520

Diagnostic Steps

• Troubleshoot with a heap dump as described in Java application "java.lang.OutOfMemoryError: Java heap space" to identify the source of retention
• To determine the contents of classloader blacklists in a heap dump, run the OQL query (see Arrays in OQL notes):

SELECT toString(e.key) FROM OBJECTS (SELECT OBJECTS s.table.@referenceArray FROM OBJECTS (SELECT OBJECTS x.@referenceArray FROM OBJECTS (SELECT OBJECTS cl.blackList.map.segments FROM org.jboss.classloader.spi.base.BaseClassLoader cl) x) s) e

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.