JBoss server slows and hangs with many threads blocked in JspServletWrapper.service()

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)

Issue

  • Our JBoss server frequently slows and hangs.  We can temporarily cut traffic to it to allow JBoss to recover.  Thread dumps during the hang show many threads blocking on a JspServletWrapper.  Stacks for the blocking threads and the thread owning the lock are like the following:
    "ajp-0.0.0.0-8009-1" daemon prio=3 tid=0x000000011af3a800 nid=0xaab waiting for monitor entry [0xfffffffdb5b5c000]
   java.lang.Thread.State: BLOCKED (on object monitor)
     at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:308)
     - waiting to lock <0xfffffffe1a0c75f8> (a org.apache.jasper.servlet.JspServletWrapper)
...

    "ajp-0.0.0.0-8009-2" daemon prio=3 tid=0x00000001026ba000 nid=0x165 sleeping[0xfffffffdc3f5b000]
    
java.lang.Thread.State: RUNNABLE
    
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
    
at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
    
at java.io.File.exists(File.java:733)
    
at org.apache.naming.resources.FileDirContext.file(FileDirContext.java:779)
    
at org.apache.naming.resources.FileDirContext.getAttributes(FileDirContext.java:435)
    
at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:747)
    
at org.apache.naming.resources.ProxyDirContext.cacheLoad(ProxyDirContext.java:1531)
    
at org.apache.naming.resources.ProxyDirContext.cacheLookup(ProxyDirContext.java:1454)
    
at org.apache.naming.resources.ProxyDirContext.lookup(ProxyDirContext.java:288)
    
at org.apache.naming.resources.DirContextURLConnection.getInputStream(DirContextURLConnection.java:366)
    
at org.apache.jasper.compiler.Compiler.isOutDated(Compiler.java:420)
    
at org.apache.jasper.compiler.Compiler.isOutDated(Compiler.java:377)
    
at org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:581)
    
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:312)
    
- locked <0xfffffffe1ab7a538> (a org.apache.jasper.servlet.JspServletWrapper)

  • Threads stall in JspServletWrapper.service with the lock owning thread making calls like the following actively servicing our servlet or JSP page:
org.apache.jsp.jsp.mypage_jsp._jspService(mypage_jsp.java:100)
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:366)
- locked <0x1939e13> (aorg.apache.jasper.servlet.JspServletWrapper)
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:322)
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:249)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

Resolution

  • Ensure pages are thread safe and do not have the following flag:

      <%@ page isThreadSafe="false" %>

    If the page truly is not thread safe, it should be recoded to be thread safe to avoid these issues.

  • Precompile JSPs to avoid recompilation in production (No longer supported as of the EAP 6, only in the EAP 5 and below.)
  • Disable development mode in jbossweb.deployer/web.xml:
       <servlet>
      <servlet-name>jsp</servlet-name>
...
      <init-param>
         <param-name>development</param-name>
         <param-value>false</param-value>
      </init-param>
...
   </servlet>
    If recompiles are needed in production, development should still be disabled and recompiles should be handled in a background thread instead, which is configured through the checkInterval parameter:
       <servlet>
      <servlet-name>jsp</servlet-name>
...
      <init-param>
         <param-name>development</param-name>
         <param-value>false</param-value>
      </init-param>
      <init-param>
         <param-name>checkInterval</param-name>
         <param-value>300</param-value>
      </init-param>
...
   </servlet>
  • Note that development mode settings from jbossweb.deployer/web.xml will not be used If a custom JSP servlet class is being used, allowing it to default to true.  In this case, development mode and other servlet settings would need to be configured in an application level web.xml along with this custom servlet class.

Root Cause

  • The threads are handling jsp recompilation checks, leading to a lot of blocked threads and contention from all the resulting disk IO when under heavy load.
  • Development mode is enabled.  Here is the relevant code from JspServletWrapper.service():
       public void service(HttpServletRequest request,
                       HttpServletResponse response,
                       boolean precompile)
            throws ServletException, IOException, FileNotFoundException {
...
           if (options.getDevelopment() || firstTime ) {
               synchronized (this) {
                  firstTime = false;
                  // The following sets reload to true, if necessary
                  ctxt.compile();

              }
       }

    

    ...
    
}
    So the thread will block in this code and have to do a recompile check if its a first time page access, or if development mode is enabled (meaning check for recompiles with every access).  Development mode adds a lot of overhead and should only be used in development.

  • A custom JSP servlet class is being used, which ignores the desired development mode settings from jbossweb.deployer/web.xml and defaults it to true.
* If you see a thread holding the JspServletWrapper lock while servicing the servlet or jsp page, check that the servlet does not implement javax.servlet.SingleThreadModel or the jsp does not have the following tag: 
    <%@ page isThreadSafe="false" %>

Such a set up forces synchronization here in JspServletWrapper to allow only one thread at a time to service the servlet:

362             if (theServlet instanceof SingleThreadModel) {
363                // sync on the wrapper so that the freshness
364                // of the page is determined right before servicing
365                synchronized (this) {
366                    theServlet.service(request, response);
367                 }
368             } else {
369                 theServlet.service(request, response);
370             }
  • Due to synchronization from development compilation settings or a single thread model servlet, threads are backing up in JspServletWrapper.service. Requests come in faster than they can be processed with this synchronization, causing thread build up until the http/ajp thread pool is exhausted, causing unresponsiveness.

Diagnostic Steps

SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.