High CPU in ContextConfig.processAnnotationsFile during Tomcat startup

Solution Verified - Updated

Environment

  • JBoss Web Server (JWS) 3.x
    • Tomcat 8.0.x

Issue

  • Tomcat start up never completes and consumes high CPU in a thread executing many recursive calls of ContextConfig.processAnnotationsFile:
3XMTHREADINFO      "localhost-startStop-1" J9VMThread:0x0000000002982900, j9thread_t:0x00007F0388015CB0, java/lang/Thread:0x00000000E05D2108, state:R, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x19, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x3C84, native priority:0x5, native policy:UNKNOWN, vmstate:R, vm thread flags:0x00000001)
3XMTHREADINFO2            (native stack address range from:0x00007F03BCC66000, to:0x00007F03BCCA7000, size:0x41000)
3XMCPUTIME               CPU usage total: 474.895982592 secs, current category="Application"
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=6137608 (0x5DA708)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at java/io/UnixFileSystem.getBooleanAttributes0(Native Method)
4XESTACKTRACE                at java/io/UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:253(Compiled Code))
4XESTACKTRACE                at java/io/File.isDirectory(File.java:860(Compiled Code))
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.processAnnotationsFile(ContextConfig.java:1989(Compiled Code))
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.processAnnotationsFile(ContextConfig.java:1989(Compiled Code))
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.processAnnotationsFile(ContextConfig.java:1989(Compiled Code))
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.processAnnotationsFile(ContextConfig.java:1989(Compiled Code))
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.processAnnotationsFile(ContextConfig.java:1989(Compiled Code))
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.processAnnotationsFile(ContextConfig.java:1989(Compiled Code))
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.processAnnotationsFile(ContextConfig.java:1989(Compiled Code))
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.processAnnotationsFile(ContextConfig.java:1989(Compiled Code))
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.processAnnotationsFile(ContextConfig.java:1989)
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.processAnnotationsUrl(ContextConfig.java:1940)
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.processAnnotations(ContextConfig.java:1898)
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.webConfig(ContextConfig.java:1149)
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.configureStart(ContextConfig.java:771)
5XESTACKTRACE                   (entered lock: org/apache/catalina/startup/ContextConfig@0x00000000E0681378, entry count: 1)
4XESTACKTRACE                at org/apache/catalina/startup/ContextConfig.lifecycleEvent(ContextConfig.java:305)
4XESTACKTRACE                at org/apache/catalina/util/LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
4XESTACKTRACE                at org/apache/catalina/util/LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
4XESTACKTRACE                at org/apache/catalina/core/StandardContext.startInternal(StandardContext.java:5110)
5XESTACKTRACE                   (entered lock: org/apache/catalina/core/StandardContext@0x00000000E05F6DB0, entry count: 2)
4XESTACKTRACE                at org/apache/catalina/util/LifecycleBase.start(LifecycleBase.java:150)

Resolution

  • Ensure the classpath does not start or end with :
  • Ensure excessive content or symlinks to it is not included under tomcat lib directories
  • If using an RPM install, update to 3.1.0+. Or if remaining on 3.0, adjust the /usr/sbin/tomcat8 file so it does not add : to the beginning of the classpath. Change:
# CLASSPATH munging
CLASSPATH="${CLASSPATH}:${CATALINA_HOME}/bin/bootstrap.jar"
CLASSPATH="${CLASSPATH}:${CATALINA_HOME}/bin/commons-logging-tomcat-juli.jar"
CLASSPATH="${CLASSPATH}:${CATALINA_HOME}/bin/tomcat-juli.jar"
CLASSPATH="${CLASSPATH}:${CATALINA_HOME}/bin/commons-daemon.jar"

to:

# CLASSPATH munging
if [ ! -z "$CLASSPATH" ] ; then
  CLASSPATH="$CLASSPATH":
fi
CLASSPATH="${CLASSPATH}${CATALINA_HOME}/bin/bootstrap.jar"
CLASSPATH="${CLASSPATH}:${CATALINA_HOME}/bin/commons-logging-tomcat-juli.jar"
CLASSPATH="${CLASSPATH}:${CATALINA_HOME}/bin/tomcat-juli.jar"
CLASSPATH="${CLASSPATH}:${CATALINA_HOME}/bin/commons-daemon.jar"
  • If seeing this on 3.1.0+, then edit /usr/libexec/tomcat8/preamble and change:
CLASSPATH="${CLASSPATH}:$(build-classpath commons-daemon 2>/dev/null)"

to:

CLASSPATH="${CLASSPATH}:${CATALINA_HOME}/bin/commons-daemon.jar"

Root Cause

  • Content from bz.apache.org is not included.ASF BZ-57823
  • The classpath starts with : and the user.dir (current directory) is set to / or some other directory with a large amount of nested content.
  • If the classpath starts with :, then it has an empty entry that the JVM handles as the current directory. Thus, ContextConfig.processAnnotationsFile will recursively scan this directory and experience high CPU if there is an excessive amount of content to traverse.
  • A sym link to a large directory is included under tomcat lib directories
  • JWS 3.0 service scripts may inherently add : to the beginning of the classpath if the CLASSPATH environment variable is empty.
  • Content from issues.jboss.org is not included.JWS-949

Diagnostic Steps

  • Capture thread dumps from start up and check the user.dir and classpath values
  • Use byteman to confirm the files being scanned.
  1. Download the latest Byteman release from http://byteman.jboss.org/downloads.html
  2. Unzip byteman into install directory .
  3. Create the Byteman rules file ContextConfig.btm with the content below:
RULE processAnnotationsFile
CLASS org.apache.catalina.startup.ContextConfig
METHOD processAnnotationsFile
AT ENTRY
IF TRUE
DO traceStack("--------------------------->processAnnotationsFile " + $1 + " " + $2 + "\n")
ENDRULE
  1. Enable the Byteman instrumentation by adding the following to your tomcat JVM options:
-javaagent:/lib/byteman.jar=script:/examplescript.btm,sys:/lib/byteman.jar
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.