JVM crash inside of libjvm.so

Solution Verified - Updated

Environment

  • OpenJDK
  • Oracle JDK

Issue

  • The JVM crashed, and the fatal error log shows libjvm.so in the problematic frame and/or top of stack.

For example:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fda54a26458, pid=820996, tid=0x00007fda501d6700
#
# JRE version: OpenJDK Runtime Environment (8.0_262-b10) (build 1.8.0_262-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.262-b10 mixed mode linux-amd64 )
# Problematic frame:
# V  [libjvm.so+0x666458]
...
Stack: [0x00007fda500d6000,0x00007fda501d7000],  sp=0x00007fda501d5880,  free space=1022k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x666458]

Resolution

  • See relevant issues listed in Root Cause.
  • If Oracle JDK;
    • Reproduce on OpenJDK.
    • Contact Oracle support.

Root Cause

Diagnostic Steps

  • Look out for hs_err_<pid> file for analysis, and check Problematic frame there:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00002ad44b961a11, pid=14746, tid=1202792768
#
# JRE version: 6.0_20-b02
# Java VM: Java HotSpot(TM) 64-Bit Server VM (16.3-b01 mixed mode linux-amd64 )
# Problematic frame:
# V  [libjvm.so+0x3b0a11]
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0000000050750000):  JavaThread "Incoming-13,127.0.0.1:8080" [_thread_in_vm, id=14912, stack(0x0000000047a12000,0x0000000047b13000)]
[...]
Stack: [0x0000000047a12000,0x0000000047b13000],  sp=0x0000000047b103c0,  free space=3f80000000000000018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x3b0a11]
V  [libjvm.so+0x3d0482]
j  org.jgroups.stack.Retransmitter.add(JJ)V+17
j  org.jgroups.stack.AckSenderWindow.add(JLorg/jgroups/Message;)V+21
j  org.jgroups.protocols.UNICAST.down(Lorg/jgroups/Event;)Ljava/lang/Object;+419
j  org.jgroups.protocols.pbcast.STABLE.down(Lorg/jgroups/Event;)Ljava/lang/Object;+114
[...]

  • Reproduce the issue with one of the following JVM options enabled: -Xcheck:jni or -XX:CheckJNICalls. This will list the native calls which may be causing the jvm crash. Output is sent to standard out.

  • Reproduce the issue with a fast/slow debug java build and check the fatal error log for a VM assert message. Note that debug builds introduce some overhead, so the application is likely to run slower.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.