High CPU or many blocked threads in BaseClassLoader.getResourcesLocally

Solution Verified - Updated 7 Aug 2024

Environment

JBoss Enterprise Application Platform (EAP) 5

Issue

After migrating to EAP 5, we see high CPU and the high CPU threads all appear to be executing BaseClassLoader.getResourcesLocally:

    Thread: ajp-0.0.0.0-8009-1 : priority:5, demon:true, threadId:408, threadState:RUNNABLE
        java.io.WinNTFileSystem.getLastModifiedTime(Native Method)
        java.io.File.lastModified(File.java:826)
        org.jboss.virtual.plugins.context.zip.ZipFileWrapper.getLastModified(ZipFileWrapper.java:140)
        org.jboss.virtual.plugins.context.zip.ZipWrapper.hasBeenModified(ZipWrapper.java:71)
        org.jboss.virtual.plugins.context.zip.ZipEntryContext.checkIfModified(ZipEntryContext.java:788)
        - locked <0x6c691ec4> (a org.jboss.virtual.plugins.context.zip.ZipEntryContext)
        org.jboss.virtual.plugins.context.zip.ZipEntryContext.getChild(ZipEntryContext.java:830)
        org.jboss.virtual.plugins.context.zip.ZipEntryHandler.createChildHandler(ZipEntryHandler.java:195)
        org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.java:690)
        org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:169)
        org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
        org.jboss.virtual.VirtualFile.getChild(VirtualFile.java:492)
        org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getResources(VFSClassLoaderPolicy.java:525)
        org.jboss.classloader.spi.base.BaseClassLoader$4.run(BaseClassLoader.java:755)
        java.security.AccessController.doPrivileged(Native Method)
        org.jboss.classloader.spi.base.BaseClassLoader.getResourcesLocally(BaseClassLoader.java:751)
        org.jboss.classloader.spi.base.BaseClassLoader.getResourcesLocally(BaseClassLoader.java:731)
        org.jboss.classloader.spi.base.BaseClassLoaderDomain.getResources(BaseClassLoaderDomain.java:520)
        org.jboss.classloader.spi.base.BaseClassLoaderDomain.getResources(BaseClassLoaderDomain.java:1175)
        org.jboss.classloader.spi.base.BaseClassLoader.loadResources(BaseClassLoader.java:554)
        org.jboss.classloader.spi.base.BaseClassLoader.findResources(BaseClassLoader.java:539)
        java.lang.ClassLoader.getResources(ClassLoader.java:1040)

We have many threads blocking on the same ZipEntryContext objects through calls to BaseClassLoader.getResourcesLocally():

    Thread: ajp-0.0.0.0-8009-1 : priority:5, demon:true, threadId:409, threadState:BLOCKED
    
    
    - waiting to lock <0xe01e3acba3534528> (a org.jboss.virtual.plugins.context.zip.ZipEntryContext)
     at org.jboss.virtual.plugins.context.zip.ZipEntryContext.getChild(ZipEntryContext.java:830)
     at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.createChildHandler(ZipEntryHandler.java:195)
     at org.jboss.virtual.plugins.context.AbstractVirtualFileHandler.structuredFindChild(AbstractVirtualFileHandler.java:690)
     at org.jboss.virtual.plugins.context.zip.ZipEntryHandler.getChild(ZipEntryHandler.java:169)
     at org.jboss.virtual.plugins.context.DelegatingHandler.getChild(DelegatingHandler.java:107)
     at org.jboss.virtual.VirtualFile.getChild(VirtualFile.java:492)
     at org.jboss.classloading.spi.vfs.policy.VFSClassLoaderPolicy.getResources(VFSClassLoaderPolicy.java:525)
     at org.jboss.classloader.spi.base.BaseClassLoader$4.run(BaseClassLoader.java:755)
     at java.security.AccessController.doPrivileged(Native Method)
     at org.jboss.classloader.spi.base.BaseClassLoader.getResourcesLocally(BaseClassLoader.java:751)
     at org.jboss.classloader.spi.base.BaseClassLoader.getResourcesLocally(BaseClassLoader.java:731)
     at org.jboss.classloader.spi.base.BaseDelegateLoader.getResources(BaseDelegateLoader.java:152)
     at org.jboss.classloader.spi.filter.FilteredDelegateLoader.getResources(FilteredDelegateLoader.java:160)
     at org.jboss.classloader.spi.base.BaseClassLoaderDomain.getResourcesFromExports(BaseClassLoaderDomain.java:791)
     at org.jboss.classloader.spi.base.BaseClassLoaderDomain.getResources(BaseClassLoaderDomain.java:507)
     at org.jboss.classloader.spi.base.BaseClassLoaderDomain.getResources(BaseClassLoaderDomain.java:1175)
     at org.jboss.classloader.spi.base.BaseClassLoader.loadResources(BaseClassLoader.java:554)
     at org.jboss.classloader.spi.base.BaseClassLoader.findResources(BaseClassLoader.java:539)
     at java.lang.ClassLoader.getResources(ClassLoader.java:1041)

Resolution

Upgrade to EAP 5.2.0 when available as this should be resolved here:
- Content from issues.jboss.org is not included.https://issues.jboss.org/browse/JBPAPP-7610
One-offs are currently available for the following versions
- EWP 5.1.0: https://access.redhat.com/jbossnetwork/restricted/softwareDetail.html?softwareId=9633
- EAP 5.1.1: https://access.redhat.com/jbossnetwork/restricted/softwareDetail.html?softwareId=9623

Root Cause

Content from issues.jboss.org is not included.JBCL-181
Compared to BaseClassLoader.getResources(), getResourcesLocally is much less performant. The main notable difference between these implementations is that getResource()/getResourceLocally() stores results in cache and loads from here first before hitting the VFS as seen from this except of getResourceLocally():
```
            if (resourceCache != null)
            {
               URL url = resourceCache.get(name);
               if (url != null)
               {
                  if (trace)
                     log.trace(this + " got resource from cache " + name);
                  return url;
               }
            }
```
getResources()/getResourcesLocally() has no such caching implemented so it goes straight to the VFS each time, which when under load can lead to high CPU from heavy filesystem access or heavy contention on locks.

Diagnostic Steps

Troubleshoot with thread dumps and CPU utilization data as described in Java application high CPU to identify high CPU consumers and problematic blocking threads.

SBR

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.