Java crash in "CompilerThread" or "JIT Compilation Thread"

Solution Verified - Updated 8 May 2025

Environment

Sun JDK
OpenJDK
IBM JDK

Issue

We have crashed our JBoss servers three times due to memory segment faults.
JBoss is crashing and producing a fatal error and core dump.

The beginning of Sun JDK/OpenJDK fatal error log shows the following:

  #
  # An unexpected error has been detected by HotSpot Virtual Machine:
  #
  #  SIGSEGV (0xb) at pc=0x00002aaaaafe08b6, pid=9053, tid=1093802304
  #
  # Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0_19-b02 mixed mode)
  # Problematic frame:
  # V  [libjvm.so+0x5248b6]
  #
  
  ---------------  T H R E A D  ---------------
  
  Current thread (0x00002aab84d33400):  JavaThread "CompilerThread1" daemon [_thread_in_native, id=9073]
  
  siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x0000000000000008

The IBM javacore shows the following:

  1XMCURTHDINFO  Current Thread Details
  NULL           ----------------------
  3XMTHREADINFO      "JIT Compilation Thread" TID:0x30EE1D00, j9thread_t:0x30116E4C, state:R, prio=10
  3XMTHREADINFO1            (native thread ID:0x3700b7, native priority:0xb, native policy:UNKNOWN)

Resolution

Upgrade to the latest JVM update to acquire any relevant fixes.
Remove or fix the offending instrumentation agent.
Disable debugging instrumentation.
Involve the JVM vendor to fix any JIT compiler bugs.

Open JDK / Sun JDK

Exclude the identified method(s) from being compiled with the following JVM option:
```
  -XX:CompileCommand=exclude,my/pkg/MyClass,theMethod
```

IBM JDK

Decrease the JIT compiler optimization level for the identified method(s). The optimization levels are, in decreasing order: scorching, veryHot, hot, warm, cold, noOpt. For example:
```
  -Xjit:{/lang/Math.max(II)I}(optLevel=warm,count=0)   
```
If the failure still occurs at the noOpt optimization level, exclude the identified method(s) from being compiled. For example:
```
  -Xjit:exclude={java/lang/Math.max(II)I}
```

Root Cause

One typical cause is JVM instrumentation code that has a bug or exposes a JVM compiler bug. Instrumentation modifies Java byte code by injecting byte code into methods. The byte code can work fine when executed by the Java interpreter, but when the JVM sees that the method has been executed multiple times and tries to optimize the code by compiling it into native code with the JIT compiler, the JVM crashes due to a JIT compiler or instrumentation code bug.

The following are sources of instrumented code:

Diagnostic and monitoring tools (e.g. Wiley).
Debugger
Aspect Oriented Programming (AOP)

Other causes:

Diagnostic Steps

General

Test with any instrumentation agents and/or debugging instrumentation disabled.
Check if the failing method matches an AOP pointcut. The issue could be related to AOP byte code injection.
Test with the latest JVM and/or version to see if it is a JIT compiler bug that has been addressed in a later release.

Open JDK / Sun JDK

Analze the Java fatal error log. The fatal error log will be created in the director defined by the user.dir environment variable in the JBoss boot.log (typically JBOSS_HOME/bin/). It will be a file named hs_err_pid<pid>.log (where <pid> is the JBoss process id).

Verify the fatal error long shows the curent thread is a CompilerThread. For example:

  Current thread (0x00002aab84d33400):  JavaThread "CompilerThread1" daemon [_thread_in_native, id=9073]

Verify the Current CompileTask shows a Java class and method. For example:

  Current CompileTask:
  opto:166% !   org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V @ 61 (147 bytes)

Check the JVM options listed in the fatal error log to see if any instrumenting agents are being specified with the -javaagent option. For example:
```
  -javaagent:/opt/jboss/wily/Agent.jar
```
Check the JVM options in the fatal error log to see if any debugging instrumentation is enabled. For example:
-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n
Test excluding the method indicated in the Current CompileTask in the fatal error log from being compiled with the following JVM option:
```
  -XX:CompileCommand=exclude,my/pkg/MyClass,theMethod
```

Specifically when facing C2 compilation issues consult the solution Solving OpenJDK C2 compilation issues, which present a few workarounds and precise diagnostic steps.

IBM JDK

See Java application down due to JVM crash for how to analyze the output from the various dump agents.

To determine the failing method, check the jextract output for the Method_being_compiled. For example:

  <gpf failingThread="0x30ee1d00" nativeFailingThread="0x269021" >
  ...
  Module=/psilocal/java160_tms/jre/lib/ppc/libj9jit24.so&#xA;
  Module_base_address=D4B1B000&#xA;&#xA;
  Method_being_compiled=java/lang/Math.max(II)I&#xA;
  </gpf>

Test excluding the failing method from being compiled with the following JVM option:
```
  -Xjit:exclude={java/lang/Math.max(II)I}
```

SBR

JVM & Diagnostics

Product(s)

Components

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.