JVM is deadlocked in safepoint pause due to cyclance

Environment

• Java
• JBoss Enterprise Application Platform (EAP)
• Cylance

Issue

• We run java applications in a system with cylance. We see the JVM reach a persistent hang and even a typical thread dump cannot be captured from it. Native traces show the VMThread continually waiting in SafepointSynchronize::begin():

#0  0x00007f71f973e945 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f71f86397e3 in os::PlatformEvent::park() () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#2  0x00007f71f85f9748 in Monitor::ILock(Thread*) () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#3  0x00007f71f85f98e1 in Monitor::lock(Thread*) () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#4  0x00007f71f86d7796 in SafepointSynchronize::begin() () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#5  0x00007f71f87e8eaf in VMThread::loop() () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#6  0x00007f71f87e9330 in VMThread::run() () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#7  0x00007f71f8640568 in java_start(Thread*) () from /opt/java/jdk/jre/lib/amd64/server/libjvm.so
#8  0x00007f71f8d2473f in ?? () from /opt/cylance/desktop/cymemdeflinux_x64.so
#9  0x00007f71f973add5 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f71f904ab3d in clone () from /lib64/libc.so.6

• And another thread is persistently stuck in malloc lock calls from cylance memory defense:

#0  0x00007f618d4ec82c in __lll_lock_wait_private () from /lib64/libc.so.6
#1  0x00007f618d4687e5 in _L_lock_16773 () from /lib64/libc.so.6
#2  0x00007f618d465833 in malloc () from /lib64/libc.so.6
#3  0x00007f618ddf5128 in tls_get_addr_tail () from /lib64/ld-linux-x86-64.so.2
#4  0x00007f618ddfaaf8 in __tls_get_addr () from /lib64/ld-linux-x86-64.so.2
#5  0x00007f618d1b826d in ?? () from /opt/cylance/desktop/cymemdeflinux_x64.so
#6  0x00007f618d461aef in sysmalloc () from /lib64/libc.so.6
#7  0x00007f618d462a3b in _int_malloc () from /lib64/libc.so.6
#8  0x00007f618d4657dc in malloc () from /lib64/libc.so.6
#9  0x00007f618ddf5128 in tls_get_addr_tail () from /lib64/ld-linux-x86-64.so.2
#10 0x00007f618ddfaaf8 in __tls_get_addr () from /lib64/ld-linux-x86-64.so.2
#11 0x00007f618d1b86d1 in ?? () from /opt/cylance/desktop/cymemdeflinux_x64.so
#12 0x00007f618dbcedd5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f618d4deb3d in clone () from /lib64/libc.so.6

Resolution

• Disable cylance

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.