JBoss EAP is slow to start
Environment
- Red Hat JBoss Enterprise Application Platform (EAP)
- 4.x
- 5.x
- 6.x
- 7.x
- Red Hat JBoss Enterprise SOA Platform (SOA-P)
Issue
- JBoss startup is slow.
- JBoss is taking almost 2 minutes to boot the server in default configuration without any applications deployed.
Resolution
- One way to help mitigate a slow startup is to slim JBoss and remove unnecessary services:
- Another option is to minimize deployments and annotation scanning:
- Check for antivirus scanning which could be slowing execution. Norton is for example notorious to slow JBoss down. Exclude the JBoss directory from being scanned.
- Check for filesystem performance issues and use a locally mounted filesystem for best performance
- See the resolutions in the referenced root cause documents below.
- Disable any antivirus that may be preventing JBoss from starting.
- Startups on EAP 5 and earlier will never exactly be fast as it is a single threaded start up process. If start times are important to you, consider upgrading to EAP 6 if you haven't already. Large efforts were made to improve start times in EAP 6 as it now offers multi threaded, concurrent start processes for much quicker start times compared to prior releases.
- For large applications in EAP 7+, generate jandex indexes beforehand per JBoss slow to start with long jandex indexing times for large applications
Root Cause
- Many jars in an ear leading to lots of annotation scanning. EAP 5 annotation scanning code is reflection based, which is very slow.
- Many EJB's in an ear, leading to lots of very inefficient dependency checking. See This content is not included.JBoss Enterprise Application Platform (EAP) takes a long time to start with several EARs.
- Slow filesystem performance
- Many *-service.xml's in the root of a sar
- EAP 5 and earlier does start up through single thread and so will never be fast
- JBoss SOA-P very slow to start after adjusting JMS destination paging parameters
- "GMS flush by coordinator at 192.168.0.1:12345 failed" in JBoss EAP 5 log
- Java garbage collection long pause times
- JBoss hangs on startup in JNDI lookup
- Java/JBoss hangs creating and authenticating connections to the database
- JBoss fails to start and deploy our applications with the main thread hanging on a socketRead through XML validation
- "/dev/random" hangs in java program when server is inactive
- JBoss is slow to start when hosting large amounts of static content or large jars with many files and packages
- Excess time spent in audit logging
- JBoss is slow to start spending much time in catalina initialization
- mod_cluster slows the JBoss start up process by trying to connect to down Apache httpd servers
- JBoss Messaging stalls startup synchronizing on an aop AspectManager
- Startup hangs with concurrent calls to LocalManagedConnectionFactory.getDriver
- This content is not included.JBoss Enterprise Application Platform (EAP) takes a long time to start with many EJBs
- JBoss EAP 6 startup/shutdown doesn't complete
- JBoss hangs at startup due to antivirus
- Spring deadlocks between DefaultListableBeanFactory and DefaultSingletonBeanRegistry
- EAP 6 start up stalls on EJB creation
- EAP 6 start up stalls in SecurityUtil.createMethodAndCacheIt
- EAP 6 deployment stalls and throws warning when validating beans.xml
- CDI enabled deployments hang in sun annotation libraries
- Web connectors sometimes never start when using WAIT_FOR_BEFORE_START
- JBoss EAP 6 stalls recursively scanning exploded deployment during startup
- Too many files in web application cause long start-up time
- JSF deployment hangs in web.xml schema validation on EAP 7
- EAP 7 start up slowness in ServiceModuleLoader.preloadModule
- JBoss EAP 6 or 7 starts slowly when one of the mod_cluster proxies is down
- EAP 7 start up with many applications using mod_cluster delayed in org.wildfly.extension.undertow.UndertowService.fireEvent
- EAP 7 deployment spends a lot of time in weld deployment processing
- EAP 7 start up hangs on logging socket handler
- Slowness / Memory issues when starting JBoss with many JSF 2 apps on java 8 where "ZipFileInflaterInputStream" count is growing and pending finalization
- Continuous high GCs and heap memory is getting accumulated with high thread contention in JBeret
- JBoss slow to start with long jandex indexing times for large applications
- JSF application start up hangs on EAP 7.4 Update 15-16
- JBoss EAP start up deadlocks on Elytron ExpressionResolver
Diagnostic Steps
General
- Review the JBoss
boot.logandserver.logand verify how long it takes for startup and if there are any associated errors or warnings. - Set JBoss logging to TRACE and compare the startup of an empty container to that with application(s) deployed.
- Enable trace for the following category and check which aspect of the deployment is taking so long
<category name="org.jboss.deployment">
<priority value="TRACE" />
</category>
- Use a profiler to determine where the majority of clock time is spent.
- Check whether NFS mount is being used. If it is, obtain information on the NFS transfer rate.
- Enable garbage collection logging and analyze it to see if there are long pauses or low throughput that explain the slowness.
- Get multiple thread dumps and look for what the "main" thread is doing over time.
Solaris / SunOS
-
Start JBoss with
trusslogging system calls to see what is happening at the system call level during the gaps in the JBoss logging. Editrun.shas follows (line 285)# Execute the JVM in the foreground truss -dleaf -o truss.txt "$JAVA" $JAVA_OPTS \ -Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" \ -classpath "$JBOSS_CLASSPATH" \ org.jboss.Main "$@"
Linux
-
Start JBoss with
stracelogging system calls to see what is happening at the system call level during the gaps in the JBoss logging. Edit run.sh as follows:# Execute the JVM in the foreground strace -f -o strace.log "$JAVA" $JAVA_OPTS \
Components
Category
Tags
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.