Why do multithreaded applications use significantly more virtual memory on a more recent version of Red Hat Enterprise Linux MALLOC_ARENA_MAX

Environment

• Red Hat Enterprise Linux (RHEL)
  • upgrade from RHEL 4 to 5
  • upgrade from RHEL 5 to 6
  • glibc

Issue

• My multithreaded application consumes a lot more memory on RHEL 6 compared to RHEL 5, or on RHEL 5 compared to RHEL 4. This can be seen from the VSZ column in top output for the process or through "Out of Memory" errors encountered in the process itself.

Resolution

This behavior can be constrained by setting MALLOC_ARENA_MAX=n to set a maximum number of malloc arenas to allocate. Set it to 1 to keep only the main_arena and hence not allocate additional address space

Note

• This solely involves virtual memory, not committed/resident memory. An application should not use appreciably more resident memory than it did under an earlier version of RHEL.
• Make sure you use glibc version of at least glibc-2.12-1.80 on RHEL 6 and at least glibc-2.5-107 on RHEL 5 because it fixes a bug wherein the environment variables MALLOC_ARENA_MAX and MALLOC_ARENA_TEST did not work as expected due to a race condition.

Root Cause

Many applications will see a large increase in virtual memory usage but not resident memory when running on RHEL 6 vs RHEL 5, or RHEL 5 vs RHEL 4. For example, This content is not included.BZ 640286.

The reason for this is that RHEL 5 and 6 include an update to glibc, which includes a new feature that creates multiple memory arenas for heap allocation. By mapping anonymous address spaces (arenas) between which threads can distribute access for heap allocation, fewer threads will be attempting concurrent access to a single arena. This generally results in a performance increase due to a reduction of lock contention between threads, but under certain workloads and memory access patterns can result in an increase in virtual memory use. This behavior is especially evident in java-based applications, as the JVM creates many threads, even for single threaded applications.

This behaviour can be controlled using environment variables MALLOC_ARENA_MAX and MALLOC_ARENA_TEST. MALLOC_ARENA_TEST specifies the number of arenas that can be created before checking for a maximum threshold (8 times the number of cores on 64-bit systems) and MALLOC_ARENA_MAX is the absolute number of arenas that should exist for the application. Both these environment variables can have a minimum value of 1.

Diagnostic Steps

Here is a simple java application reading for a string.

# cat ReadString.java
import java.io.*;

public class ReadString {
   public static void main (String[] args) {
      //  prompt the user to enter their name
      System.out.print("Enter your name: ");

      //  open up standard input
      BufferedReader br = new BufferedReader(new InputStreamReader(System.in));
  
      String userName = null;

      //  read the username from the command-line; need to use try/catch with the
      //  readLine() method
      try {
         userName = br.readLine();
      } catch (IOException ioe) {
         System.out.println("IO error trying to read your name!");
         System.exit(1);
      }

      System.out.println("Thanks for the name, " + userName);
   }
}

• Compile and run the application as below

    javac ReadString.java
    java ReadString

• While application is running, from other terminal execute the below command to check the applications virtual memory That would show the size of VSZ going to 2GB

    # ps auxwww | grep -v grep | grep java
    root      2842  0.2  1.1 1578600 17540 pts/0   Sl+  17:51   0:00 java ReadString

Open a This content is not included.new support ticket and provide the following:

• pmap from the previous version of RHEL for comparison ( if upgrading from RHEL 5 to 6, provide pmap data from RHEL 5 and 6)
• evidence of actual OOM episodes or if it's just an unexplained metric
• contents of /proc/[pid]/status

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.