ActiveMQ 6.3 runs out of direct buffer memory under load

Solution Verified - Updated

Environment

  • Red Hat JBoss Fuse
    • 6.3 R 8 and earlier

Issue

Under heavy load, with KahaDB message store, A-MQ 6.3 fails with messages about lack of direct buffer memory. A-MQ 6.2 handled the load with the same settings. The KahaDB settings include preallocationStrategy=zeros.

Exceptions of the following form might be seen:

2016-10-13 06:59:53,726 | WARN  | ActiveMQ Transport: tcp:///10.57.67.143:37979@61616 | XATransaction                    | tivemq.transaction.XATransaction   91 | 162 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630187 | Store COMMIT FAILED:
java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:693)[:1.8.0_101]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)[:1.8.0_101]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)[:1.8.0_101]
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)[:1.8.0_101]
        at sun.nio.ch.IOUtil.write(IOUtil.java:58)[:1.8.0_101]
        at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)[:1.8.0_101]
        at org.apache.activemq.store.kahadb.disk.journal.Journal.doPreallocationZeros(Journal.java:351)
        at org.apache.activemq.store.kahadb.disk.journal.Journal.preallocateEntireJournalDataFile(Journal.java
        ...

Resolution

Use a different preallocation strategy, e.g., os_kernel_copy, or upgrade to A-MQ 6.3 Roll-up 9 or later.

Root Cause

This problem is caused by bug Content from issues.jboss.org is not included.ENTMQ-1980 / Content from issues.apache.org is not included.AMQ-6451. Only the "zeros" preallocation strategy is affected. The problem was not so apparent before A-MQ 6.3, because earlier releases used heap-backed buffers for journal preallocation. There might have been additional load on the garbage collector. Despite their advantages, direct buffers have the disadvantages of being notoriously difficult to de-allocate, and leaks of direct buffers will rapidly cause a problem.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.