JVM hung from incomplete safepoint pause

Solution Verified - Updated

Environment

  • Java
  • JBoss Enterprise Application Platform (EAP)

Issue

  • Our JVM hangs and even a typical thread dump cannot be captured from it. Native traces show the VMThread continually waiting in SafepointSynchronize::begin():
#0  0x00007f71f973e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f71f86397e3 in os::PlatformEvent::park() () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#2  0x00007f71f85f9748 in Monitor::ILock(Thread*) () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#3  0x00007f71f85f98e1 in Monitor::lock(Thread*) () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#4  0x00007f71f86d7796 in SafepointSynchronize::begin() () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#5  0x00007f71f87e8eaf in VMThread::loop() () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#6  0x00007f71f87e9330 in VMThread::run() () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#7  0x00007f71f8640568 in java_start(Thread*) () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#8  0x00007f71f8d2473f in ?? () from /opt/cylance/desktop/cymemdeflinux_x64.so
#9  0x00007f71f973add5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f71f904ab3d in clone () from /lib64/libc.so.6

Resolution

  • Identify and address any other threads that are not reaching a safepoint

Root Cause

Diagnostic Steps

  • Add JVM options like the following for information on safe point pauses:
-XX:+UnlockDiagnosticVMOptions  -XX:+PrintSafepointStatistics -XX:PrintSafepointStatisticsCount=1 -XX:+LogVMOutput -XX:LogFile=<path>/vm.log
  • Capture native thread traces from the issue via pstack (provided by the gdb package) and jstack -m:
jstack -m $PID >> jstack.out
pstack $PID >> pstack.out
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.