JBoss slows after 24 hours for no apparent reason

Environment

• Red Hat JBoss Enterprise Application Platform (EAP) 6.x
• Java 7

Issue

• Our JBoss server mysteriously slows after a day or so for no apparent reason. There is no resource contention (no high memory, swapping, CPU, or heavy disk usage), no GC issues, and no threads persisting in blocks, waits or sleeps. Threads remain runnable with available CPU time but despite that everything is considerably slower.

Resolution

• Do not use -XX:+TieredCompilation. It has had Content from bugs.sun.com is not included.known performance issues.
• If running on Windows through standalone.bat, it may be setting -XX:+TieredCompilation automatically for you. Remove the following block from standalone.bat so it does not do that:

if not "%PRESERVE_JAVA_OPTS%" == "true" (
  rem Add tiered compilation, if supported (64 bit VM), and not overriden
  echo "%JAVA_OPTS%" | findstr /I "\-XX:\-TieredCompilation \-client" > nul
  if errorlevel == 1 (
    "%JAVA%" -XX:+TieredCompilation -version > nul 2>&1
    if not errorlevel == 1 (
      set "JAVA_OPTS=-XX:+TieredCompilation %JAVA_OPTS%"
    )
  )
)

• standalone.bat is fixed with -XX:+TieredCompilation removed in EAP 6.2.0+
• Certain JDK releases may Content from bugs.sun.com is not included.enable TieredCompilation by default. Set -XX:-TieredCompilation in your JVM options to override any potentially problematic JDK defaults.
• Upgrade to JDK8, where the issues with -XX:+TieredCompilation appear to have been addressed.

Root Cause

• The JVM is configured to use -XX:+TieredCompilation, which is causing performance issues in the JDK layer
• AS 7 start up scripts (standalone.sh/bat) were set up to use TieredCompilation if available
• Due to its performance issues, that was dropped from standalone.sh before EAP 6.0.0's release (Content from issues.jboss.org is not included.AS7-4547), but this wasn't dropped from standalone.bat (Content from issues.jboss.org is not included.JBEAP-26 & This content is not included.BZ-998913)

Diagnostic Steps

• Check if -XX:+TieredCompilation is present in the JVM options used by JBoss through the boot/server log start up output

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.