JBoss is slow to start spending much time in catalina initialization

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)

Issue

  • Our JBoss server is slow to start. We can specifically see that it is slow during the catalina initialization of JBoss Web as evident from these log lines:

      INFO  [org.apache.coyote.http11.Http11AprProtocol] Initializing Coyote HTTP/1.1 on http-0.0.0.0-8080
  INFO  [org.apache.coyote.ajp.AjpAprProtocol] Initializing Coyote AJP/1.3 on ajp-0.0.0.0-8009
  INFO  [org.apache.catalina.startup.Catalina] Initialization processed in 30728 ms

  • Thread dumps show the main thread in the following code:

      "main" prio=1 tid=0x0000000049c896c0 nid=0x3185 runnable [0x0000000040f2e000..0x0000000040f32d10]
	at org.apache.tomcat.jni.SSL.initialize(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.apache.catalina.core.AprLifecycleListener.initializeSSL(AprLifecycleListener.java:220)
	- locked <0x00002b0f556f6d60> (a java.lang.Class)
	at org.apache.catalina.core.AprLifecycleListener.lifecycleEvent(AprLifecycleListener.java:84)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
	at org.apache.catalina.core.StandardServer.initialize(StandardServer.java:767)
	at org.apache.catalina.startup.Catalina.load(Catalina.java:518)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:558)

Resolution

  • If SSL is not needed for JBossWeb, then disable it in your server.xml so time is not wasted here:

      <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="off" />

  • If SSL is needed, change the SSLRandomSeed used to be "/dev/urandom" instead of "/dev/random" in your server.xml:

         <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" SSLRandomSeed="/dev/urandom" />

Root Cause

  • SSL initialization is likely waiting here for random data from "/dev/random". The default "/dev/random" generates random number based on noise or entropy (keyboard/mouse activity, etc.) on the system; the higher the noise the faster the output. If there is no noise this tool won't return any data and will block until it can return data based off the system entropy and so threads and processes using it can stall for extended periods.
  • This content is not included."/dev/random" hangs in java program when server is inactive

Diagnostic Steps

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.