JBoss slow to start with long jandex indexing times for large applications
Environment
- JBoss Enterprise Application Platform (EAP) 7.4.6+
Issue
- We have large applications deployed on EAP 7. We've seen that its start up time and heap required during start up has increased moving to EAP 7.3.1+ and again moving to 7.4.6+. When the app first begins its start up, the log stays silent for many minutes. During that time, CPU remains high on a single core from a single thread and thread dumps in that time show continual jandex indexing:
"MSC service thread 1-7" #22 prio=5 os_prio=0 cpu=1622528.18ms elapsed=1799.27s tid=0x0000000003561800 nid=0x9a0d runnable [0x00007efda0380000]
java.lang.Thread.State: RUNNABLE
at org.jboss.jandex.StrongInternPool.eq(StrongInternPool.java:139)
at org.jboss.jandex.StrongInternPool.intern(StrongInternPool.java:246)
at org.jboss.jandex.NameTable.intern(NameTable.java:109)
at org.jboss.jandex.ClassInfo.setMethods(ClassInfo.java:781)
at org.jboss.jandex.Indexer.index(Indexer.java:1992)
at org.jboss.as.server.deployment.annotation.ResourceRootIndexer.indexResourceRoot(ResourceRootIndexer.java:99)
at org.jboss.as.server.deployment.annotation.AnnotationIndexProcessor.deploy(AnnotationIndexProcessor.java:51)
at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:182)
- locked <0x00000004054bcef8> (a org.jboss.as.server.deployment.DeploymentUnitPhaseService)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.lang.Thread.run(java.base@11.0.5/Thread.java:834)
Resolution
- Scale up heap as needed for any increased indexing overhead.
- For notable start up increases to avoid jandex indexing time at start up, generate indexes for your application jars beforehand as described in point 1 of How to create a jandex index for a jar so that JBoss EAP 6 or EAP 7 modules will scan a modules annotations
- If deploying many of the same method, update to EAP 7.4.14+ for jandex improvements handling this. As a workaround, pre-index the app with jandex 2.0.5 for a shorter deployment time and shorter pre-index time.
Root Cause
- EAP 7.3.1+ upgrades from jandex 2.0.5 to 2.1.2. EAP 7.4.6+ then upgrades to jandex 2.4.2. Some jandex indexing increase is likely unavoidable in these later versions as it has newer fixes to provide more features requiring more/longer scanning and bigger indexes. Like Content from github.com is not included.this change in 2.1.2 and Content from github.com is not included.this change in 2.4.2.
- If a jar contains lots of the same method name across different classes, this can hit additional indexing time:
- Additional heap overhead improvements for large deployment cases:
Diagnostic Steps
- In a heap dump, use the following OQL to look for large StrongInternPools. Inspect the largest for duplicates indicative of This content is not included.JANDEX-50
SELECT size FROM org.jboss.jandex.StrongInternPool
SBR
Components
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.