JVM crash when executing compiled java code

Environment

• Java

Issue

• The JVM crashed and provided a hs_err crash file indicates the crash occurred in some compiled Java code:

Stack: [0x0000000011680000,0x0000000011780000],  sp=0x000000001177e6c0,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J  org.crashing.compiled.Class.method()J
...

• Crash in code compiled by the C1 compiler:

Stack: [0x00007f01aae5e000,0x00007f01aaf5f000],  sp=0x00007f01aaf5d240,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 35615 C1 sun.nio.ch.Util.free(Ljava/nio/ByteBuffer;)V (13 bytes) @ 0x00007f06294973ce [0x00007f0629497360+0x6e]

• Crash in code compiled by the C2 compiler:

Stack: [0x00003ffbdb7a0000,0x00003ffbdb9a0000],  sp=0x00003ffbdb99ba20,  free space=2030k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 1985 C2 java.net.URLStreamHandler.parseURL(Ljava/net/URL;Ljava/lang/String;II)V (1193 bytes) @ 0x00003fff71efdc3c [0x00003fff71efc280+0x19bc]

Resolution

• Upgrade to the latest JDK update to consume any relevant compiler fixes.
• As a workaround, exclude the problem method from compilation, for example:

-XX:CompileCommand=exclude,org/crashing/compiled/Class,method

Root Cause

• A JDK compiler bug is resulting in some bad code generation
• JVM crash in JIT-compiled org.apache.http.impl.cookie.BestMatchSpec.formatCookies() method
• JVM crash in com.sun.crypto.provider.GCTR.doFinal
• JVM crash in JIT-compiled org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.done() method
• Java crash in java.net.URLStreamHandler.parseURL C2 compiled Java code
• JVM crash in String.compareTo(Object) on AVX2 processor

Diagnostic Steps

Note if the crash occurred in code compiled by the C1 or C2 compiler. Tiered compilation is enabled by default in JDK8+.

Java bytecode is first compiled by the C1 compiler, then again by the C2 compiler based on the number of invocations and heuristics. This is done to optimize the system resources (cpu, memory) required for Just in Time (JIT) compiling.

C1 compiling takes less time/resources but creates less optimized code. C2 compiling takes more time/resources and memory but creates better optimized code.

Check if there is any instrumentation doing bytecode manipulation. If possible, reproduce the issue with the instrumentation removed.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.