Interpreting JVM crash file in OpenJDK/OracleJDK

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL)
    • 7.x
  • OpenJDK 8
  • OpenJDK 11

Issue

  • What is the interpretation of JVM crash file output, hs_err_pid.log file?
  • How to interpret the hs_error file?
  • What does the Fatal Error Log contain?
  • Are the debug info installed?

Resolution

Disclaimer: Links contained herein to an external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

As described on the solution Java application down due to JVM crash , when there is a JVM crashed a hs_err file will be generated indicates the crash occurred the JVM itself. By default, the crash file is called hs_error_pidXX.log, where XX is the Java process PID and it will be created in the working directory of the process.

For installing the debug info, see solution: Verify OpenJDK has debug symbols installed.

This can be changed using the flag:

   java -XX:ErrorFile=/dir/example/log/another_error_name%p.log

An example of Java HotSpot(TM) 64-Bit crash file1:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f3f2590b44f, pid=3277, tid=139909094418176
#
# JRE version: 7.0_06-b24
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.2-b09 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# V  [libjvm.so+0x75744f]  ParMarkBitMap::mark_obj(HeapWord*, unsigned long)+0xdf
#
# Core dump written. Default location: //core or core.3277 (max size 1 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again
#
[...]
---------------  T H R E A D  ---------------

Current thread (0x0000000000620800):  GCTaskThread [stack: 0x00007f3f1fcfa000,0x00007f3f1fdfb000] [id=3283]
[...]
Stack: [0x00007f3f1fcfa000,0x00007f3f1fdfb000],  sp=0x00007f3f1fdf9c20,  free space=1023k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x75744f]  ParMarkBitMap::mark_obj(HeapWord*, unsigned long)+0xdf
V  [libjvm.so+0x79eb3e]  PSParallelCompact::MarkAndPushClosure::do_oop(oopDesc**)+0xce
V  [libjvm.so+0x5a1a60]  JNIHandles::oops_do(OopClosure*)+0x20
V  [libjvm.so+0x77a1b9]  MarkFromRootsTask::do_it(GCTaskManager*, unsigned int)+0xd9
V  [libjvm.so+0x4b74c0]  GCTaskThread::run()+0x130
V  [libjvm.so+0x746a00]  java_start(Thread*)+0x100
[...]

The crash file is divided into sections:

  • Header Section
  • Thread Section: --------------- T H R E A D ---------------
  • Process Section: --------------- P R O C E S S ---------------
  • System Section: --------------- S Y S T E M ---------------

Header Section


At the beginning of the `hs_error` file, the header will contain a short description of the problem.
The first line `A fatal error has been detected by the Java Runtime Environment` explains an error occurred and it caused the crash.
Sectioncorresponding meaning
SIGSEGV (0xb) at pc= ExampleA segmentation error occurred
JRE version: 7.0_06-b24JRE version, on this case 1.7.06-b24
Java VM: Java HotSpot(TM)Oracle HotSpot or OpenJDK
Problematic framethe problematic frame of the error
Core dump writtenA core dump can be generated or not, and its location

Problematic frame interpretation: J=compiled Java code, j=interpreted, Vv=VM code, C=native code.

Where the crash usually occurs for the following reasons:

Crash inIssue reason
Crash in Native CodePossible bug in native code or the Java Native Interface (JNI) library code.
Crash in Compiled CodePossibly a compiler bug that has resulted in incorrect code generation.
Crash in HotSpot Compiler ThreadBug on the compiler itself
Crash in VM ThreadGC issue like heap corruption
Crash Due to Stack OverflowStack is being written past the end of the stack
AbbreviationsMeaning
"V"VM code (from C++ code)
"v"VM code (generated by VM at runtime)

The core dump will be generated on a default location and this is disclaimed here::
Core dump written. Default location: //core or core.3277 (max size 1 kB).

Note without the debug some information might be hard to interpret, see Verify OpenJDK has debug symbols installed on the source machine.

Finally, there will be a link to submit a bug report, usually: http://bugreport.sun.com/bugreport/crash.jsp
The problematic frame will point to the possible cause and it a hint to search if this happened previously. Core dump can be use for investigations on native usage, but one can use VM.info for

Thread Section


The second section is the `Thread section` and it will show the current thread and the stack.
Where possible, the next output in the error log is the thread stack.
Sectioncorresponding meaning
Current thread Current stack pointer
StackIt shows the entire stack

Process Section


The third section contains information about the whole process, including the thread list and memory usage of the process.

System Section


The final section brings the system information and as such, the output will depend on the operating-system but generally, it will include: OS version, CPU information, and a memory configuration summary.

Additional information can be found in Content from www.oracle.com is not included.Oracle Appendix C - Fatal Error Log and Content from docs.oracle.com is not included.Oracle Troubleshoot Guide.

1

Note this is a HotSpot 64-bit JVM, because it has 64-Bit Server VM. A 32-bit machine would only bring Java HotSpot(TM) Server VM. A 64bit OpenJDK brings the string OpenJDK 64-Bit Server VM and 32-bit brings OpenJDK Server VM.

Root Cause

There are several reasons for a crash, the following solutions describe some:

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.