JBoss is slow to start when hosting large amounts of static content or large jars with many files and packages
Environment
- JBoss Enterprise Application Platform (EAP) 4.3
Issue
- JBoss is very slow to start when hosting large amounts of static content or large jars with many packages
-
The logging during this time shows only the following types of "Indexed pkg" TRACE messages:
TRACE [org.jboss.mx.loading.ClassLoaderUtils] Indexed pkg: ROOT.war.images, UCL: org.jboss.mx.loading.UnifiedClassLoader3@1ed54a0{ url=file:/path/to/jboss-as/server/default/deploy/jboss-web.deployer/ ,addedOrder=6} - The thread dumps show the main thread continuously hitting the filesystem during this time:
"main" prio=1 tid=0x081481b8 nid=0x7f74 runnable [0x301ca000..0x301ccfa0] at java.io.UnixFileSystem.list(Native Method) at java.io.File.list(File.java:937) at java.io.File.listFiles(File.java:1093) at org.jboss.mx.loading.ClassLoaderUtils$FileIterator.getNextEntry(ClassLoaderUtils.java:501) at org.jboss.mx.loading.ClassLoaderUtils$ClassPathIterator.getNextEntry(ClassLoaderUtils.java:618) at org.jboss.mx.loading.ClassLoaderUtils.updatePackageMap(ClassLoaderUtils.java:320) at org.jboss.mx.loading.ClassLoaderUtils.updatePackageMap(ClassLoaderUtils.java:254) at org.jboss.mx.loading.UnifiedLoaderRepository3.updatePackageMap(UnifiedLoaderRepository3.java:813)
Resolution
-
If this issue is caused by jars, you can see that the loop will actually end if it sees a META-INF/INDEX.LIST file and instead call readJarIndex. Provide an INDEX.LIST and readJarIndex will then read through just this file and create indexes as the file specifies rather than having to determine these dynamically through heavy filesystem access.
- If the issue is caused by static content deployed with applications, trying hosting the static content on JBoss instead through this manner by editing your server.xml:
<Host name="localhost" ...> <!-- ADD external DIRECTORY --> <Context path="/staticContent" docBase="/path/to/staticContent" reloadable="true"> </Context> <!-- The rest of your Host entity --> </Host>This should make all content under /path/to/staticContent accessible through http://jbossip:port/staticContent as well. This set up does not go through any deployers and so there is no need to create the package indexes as before and thus the filesystem hits that slowed the startup before are eliminated.
Root Cause
-
Here is the relevant ClassLoaderUtils.updatePackageMap method being hit:
static String[] updatePackageMap(RepositoryClassLoader cl, Map packagesMap, ClassPathIterator cpi, HashSet pkgNameSet) throws Exception { boolean trace = log.isTraceEnabled(); ClassPathEntry entry; while( (entry = cpi.getNextEntry()) != null ) { String name = entry.getName(); // First look for a META-INF/INDEX.LIST entry if( name.equals("META-INF/INDEX.LIST") ) { readJarIndex(cl, cpi, packagesMap, pkgNameSet); // We are done break; }// Skip empty directory entries if( entry.isDirectory() == true ) continue; String pkgName = entry.toPackageName(); addPackage(pkgName, packagesMap, pkgNameSet, cl, trace); } cpi.close(); // Return an array of the package names String[] pkgNames = new String[pkgNameSet.size()]; pkgNameSet.toArray(pkgNames); return pkgNames;}
-
So here the thread is going through this while loop hitting the file system heavily as it iterates through and creates indexes for all the entries in this ClassPathIterator with the entries encompassing all the directories/files under the static content or jars.
Diagnostic Steps
- See JBoss is slow to start.
- Gather thread dumps throughout the startup process to see what the main thread is doing.
- Capture TRACE logging during start up.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.