JVM crash in org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.done() method

Solution Verified - Updated

Environment

  • Red Hat AMQ 7.2.6
  • Async I/O journal persistence
  • Red Hat Enterprise Linux (RHEL) 7
    • kernel 3.10.0-1127.8.2.el7.x86_64
    • kernel 3.10.0-1136.el7.x86_64

Issue

  • All JBoss servers in the domain are crashing with the fatal error log showing the following stack:
j  org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.done(Lorg/apache/activemq/artemis/nativo/jlibaio/SubmitInfo;)V+1
v  ~StubRoutines::call_stub
V  [libjvm.so+0x685d2b]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0xddb
V  [libjvm.so+0x6cd5c8]  jni_invoke_nonstatic(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .constprop.109]+0x208
V  [libjvm.so+0x6cf6a6]  jni_CallVoidMethod+0x166
C  [libartemis-native-64.so+0x29b6]  Java_org_apache_activemq_artemis_nativo_jlibaio_LibaioContext_blockedPoll+0x156

Resolution

The following workarounds are considerations until a fix is available:

Option 1

  • Downgrade the kernel to 3.10.0-1062.18.1.el7.x86_64

Option 2

Replace on "messaging-activemq:4.0" subsystem:

                <journal min-files="10" pool-files="10"/>

with

                <journal type="NIO" min-files="10" pool-files="10"/>

It is recommended to test the change in a non-production system.

Option 3

  1. Download the attached activemq-artemis-native.zip which contains the following files:
  • activemq-artemis-native-1.0.2.redhat-00001.jar
  • libartemis-native-linux-x86_64-1.0.2-redhat-1.so
  • libartemis-native-linux-i686-1.0.2-redhat-1.so

  1. Shutdown the broker

  2. Backup your existing distribution

  3. Replace the existing libraries with the ones included in activemq-artemis-native.zip as follows:

4.1) Rename the updated libraries in the activemq-artemis-native.zip as follows:

mv libartemis-native-linux-x86_64-1.0.2-redhat-1.so libartemis-native-64.so
mv libartemis-native-linux-i686-1.0.2-redhat-1.so libartemis-native-32.so

4.2) Copy the updated the libraries to the following location replacing the original libraries:

  • [ARTEMIS_HOME]/lib/activemq-artemis-native-1.0.2.redhat-00001.jar
  • [ARTEMIS_HOME]/bin/lib/linux-x86_64/libartemis-native-64.so
  • [ARTEMIS_HOME]/bin/lib/linux-i686/libartemis-native-32.so
  1. Restart the broker

Important Note

These options are being made available to work around a bug in the kernel. It is recommended to upgrade the kernel as soon as a fix becomes available. There will be no need to rollback option 3 once the kernel is patched as these libraries provide a safeguard that prevents the JVM crash and will be provided with future distributions of AMQ 7.

Root Cause

AMQ Broker tracker:
This content is not included.This content is not included.https://issues.redhat.com/browse/ENTMQBR-3642

Kernel tracker:
This content is not included.This content is not included.https://bugzilla.redhat.com/show_bug.cgi?id=1845326

JBoss EAP tracker:
This content is not included.This content is not included.https://issues.redhat.com/browse/JBEAP-19722 fixed in 7.3.3

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.