Netty 4.x high memory usage

Solution Unverified - Updated

Environment

  • Red Hat AMQ 7

Issue

  • Java process size is much larger than expected.
  • top output is high but heap size is normal.
  • Java process gradually grows larger until having to restart to avoid OOM killer.

Resolution

By default, Artemis will allocate native memory up to 2 times the maximum heap size via the Netty/NIO libraries callingUnsafe.allocateMemory.

Ways to limit Netty 4.x native memory usage:

  1. Limit Unsafe.allocateMemory allocations by reducing the maximum heap size (default maximum is 2x maximum heap size) or by explicitly setting the maximum with the io.netty.maxDirectMemory option (e.g. -Dio.netty.maxDirectMemory=2147483648 for 2g).

  2. Instruct netty to use java.nio direct buffer allocations instead of Unsafe.allocateMemory by setting -Dio.netty.maxDirectMemory=0. java.nio allocations are limited with the -XX:MaxDirectMemorySize option. The default value for MaxDirectMemorySize is 0. This does not mean 0 bytes. It means the JVM determines the MaxDirectMemorySize. The value can be determined by inspecting sun.misc.VM.directMemory in a heap dump. MaxDirectMemorySize can be lowered (e.g. -XX:MaxDirectMemorySize=2g) if testing indicates it can be reduced.

For Java 11: You can check jdk.internal.misc.VM on a heap dump, .

Note: Aggressive checking for memory leaks can be enabled in a test/dev environment with one of the following Java options:

-Dio.netty.leakDetection.level=advanced

or 

-Dio.netty.leakDetection.level=paranoid

This will not provide more details on memory usage, but will check for memory leakage. This option is known to adversely impact performance, so it is not recommended to do this in production except as a last resort.

Diagnostic Steps

Get a heap dump at a time when the Java process size is much larger than expected and inspect io.netty.util.internal.PlatformDependent.DIRECT_MEMORY_COUNTER for the amount of memory allocated by Netty.

Enable Java Native Memory Tracking (NMT) and search for Unsafe_Allocate and check the total under "Internal" (JDK8) or "Other" (JDK11).

SBR
Product(s)
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.