Java java.lang.OutOfMemoryError "unable to create new native thread" or "Failed to create a thread"

Solution Verified - Updated

Environment

  • OpenJDK derivative
    • Adoptium
    • Azul
    • Oracle
    • Red Hat
  • IBM JDK

Issue

  • Encountered the following exception in the JBoss server log using OpenJDK or Oracle JDK:

          java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:597)
        ...

  • The IBM javacore file shows the following:

          1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received

  • The Oracle crash file reports the following:

    # There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create GC thread. Out of system resources.
[...]

  • ActiveMQ on broker throw java OOM error

  • The following in the OpenJDK or Oracle/Sun JDK fatal error log:

    #  Out of Memory Error (gcTaskThread.cpp:46), pid=5490, tid=139954866263808

Resolution

64-bit:

If the issue happens on 64-bit, the issue is typically resolved by increasing OS limits or decreasing system resource demand:

  • Increase OS limits on the number of threads:

    • Linux

      • Set a higher ulimit for open files in /etc/security/limits.conf on Linux. For example:
              soft    nofile           1024
        hard    nofile           8192
          
        or to set both hard and soft to the same value:

        - nofile 2048

      Note: Do not set it to an overly high value, as it can cause slowness. See https://access.redhat.com/solutions/4713351.

      • Set higher ulimit for max user processes in /etc/security/limits.conf on Linux. For example:
      soft    nproc           2048
hard    nproc           8192

      • For RHEL 6, a modification to /etc/security/limits.d/90-nproc.conf is needed rather than /etc/security/limits.conf, refer to the following article for more detatails: Setting nproc in /etc/security/limits.conf has no effect in Red Hat Enterprise Linux 6.

      • If the maximum number of threads will increase beyond the default allowed (32768) you will need to increase this limit in /etc/sysctl.conf. This value is controlled by pid_max, and to double this value the following line would be added:

        kernel.pid_max = 65536

      • You may also need to update the kernel's vm.max_map_count (default 65530) if JVM thread counts are exceeding this limit, but such a high legitimate thread count is rare.

  • Decrease the number of threads.

    • If you have applications that use JSF, try the following settings in their web.xml:
      <context-param>
    <param-name>facelets.REFRESH_PERIOD</param-name>
    <param-value>-1</param-value>
  </context-param>
  <context-param>
    <param-name>facelets.DEVELOPMENT</param-name>
    <param-value>false</param-value>
  </context-param>

  • Avoid running multiple copies of the same application by ensuring it's fully stopped before restarting.

  • In OCP 4.X, pls ref Unable to create more than 1024 Threads in OCP 4

  • In OCP 3.X, pls ref Introducing 'default pid limit' within docker to improve reliability of OpenShift Container Platform

  • Resolve downstream issues causing threads to build up or hang (e.g. a DB connection limit/bottleneck).

32-bit:

This issue is most commonly resolved by moving to 64-bit, where there is no memory split and no 4GB limit for address space.

If moving to 64-bit is not an option, the following are ways to avoid the issue on 32-bit:

Root Cause

Diagnostic Steps

  • Is the process started with systemd? systemd has its own limits and ulimits are ignored. See How to set limits for services in RHEL and systemd.

  • See "java.lang.OutOfMemoryError: unable to create new native thread" in Openshift Enterprise for troubleshooting applications deployed on OpenShift.

  • Analyze the OpenJDK (Adoptium, Oracle, Red Hat, etc.) fatal error log. Note: If a fatal error log is not created, the application is catching OutOfMemoryError.

  • Determine the amount of available physical memory, whether 32-bit or 64-bit, and the memory split for 32-bit.

    • Linux: Capture meminfo output to see available memory, whether 32 or 64-bit, and the memory split for 32-bit. On 64-bit systems HighTotal  and HighFree will be 0. On 32-bit systems, LowTotal will indicate the memory split (e.g. if it is approx 1GB, it is a 3/1 split).

                cat /proc/meminfo > meminfo.txt

    • Solaris: Run the attached This content is not included.meminfo_solaris.sh script. Output will be sent to meminfo.out.

        bash ./meminfo_solaris.sh

  • If 32-bit, determine address space allocation from the fatal error log:

    • vi hs_err_pid<pid>.log and remove heading and tailing logs and only leave mmap data in the "Dynamic libraries" section.
    • awk '{ print $1 }' hs_err_pid<pid>.log > mmap.txt
    • Create the following script run.sh:
        #!/bin/bash

    awk $([[ $(awk --version) = GNU* ]] && echo --non-decimal-data) -F- '
        BEGIN {OFS = FS; sum = 0}
        {
            $1 = sprintf("%d", "0x" $1)
            $2 = sprintf("%d", "0x" $2)
            sum = sum + ($2 - $1)
        } END { printf "%d bytes, %d MiB\n", sum, sum / (1024 * 1024) }' mmap.txt
    • Run the script run.sh and check if the address space allocated is next the maximum for the OS and 32-bit. For example:
    $ sh ./run.sh 
4294889472 bytes, 4095 MiB

  • Is the Java application running in a dedicated environment, or are there other applications (e.g. colocated JBoss instances) and/or services (e.g. a database) competing for available memory?

  • Is the Java application running on dedicated physical hardware, or on a guest OS in a virtual  environment?

  • What JVM options are being used

  • Enable garbage collection logging and analyze it to see if the maximum heap size and/or perm gen space can be decreased.

  • Check if the thread stack size is being explicitly set. The default thread stack size is typically much larger than needed. See Java default thread stack size.

  • Get a thread dump to see if something is spawning threads uncontrollably.

  • Determine the limits for the process. For example, on Linux: cat /proc/<pid>/limits > limits.txt. Check the "soft values".

  • Linux/Solaris: To see if the "max user processes" limit is being exceeded, compare ulimit -u  output (run as the user running JBoss) to the number of threads in the thread dump to see if the max user processes limit is being exceeded.

    • ps -eLf output can easily show all running threads and processes on RHEL. Count how many threads are under the JBoss user. For example: cat ps.txt | grep "^jbossas" | wc -l

    • Check the sosreport for the number of threads for the Java user: For example:

      grep "jbossas" sosreport/sos_commands/process/ps_auxwwwm | wc -l
1029

      Note: The result will be slightly larger number than the actual number of threads due to the nature of ps_auxwwwm output, where every process has a summary entry, so even a process with a single thread has 2 entries.

  • The fatal error log will show the nproc value:

    rlimit: STACK 10240k, CORE 0k, NPROC 1024, NOFILE 4096, AS infinity
    • Check the nproc value of the running process: cat /proc/<JBOSS_PID>/limits. What does it show for "Max processes"? If it is different than expected, the configuration values are not being picked up. The limits are inherited from your interactive login (i.e., your shell process). To ensure the latest settings, start a new login shell as that user and ensure you see what you want with cat /proc/$$/limits ("$$" gets expanded to your current shell PID). See How to set limits (ulimit) for services in RHEL6 and older with upstart, sysvinit.

  • Linux/Solaris: Compare ulimit -n and netstat output to see if the number of sockets is exceeding the open file limit.

  • Test if ulimit settings need to be changed or added by compiling the attached This content is not included.ThreadTest class on the box running the Java application (javac ThreadTest.java), then run it as the user that the Java application is run under, passing in values such as 1500, 3000,  5000, and 10,000 and using the same JVM options used to start the Java applicaton. Does the script complete? If not, what number does it stop on? For  example:

          java -server -Xss128k -Xms1024M -Xmx1024M -XX:PermSize=128M -XX:MaxPermSize=128M -XX:+UseParallelGC -XX:ParallelGCThreads=4 -XX:+UseParallelOldGC -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 ThreadTest 5000

  • Test running the Java application as root to determine if the issue is ulimit related, as by default everything for root is unlimited.

  • Use the following formula to determine approximately how many threads can be created:

          Number of Threads = (MaxProcessMemory -  JVMMemory - ReservedOsMemory - OtherPrograms) / (ThreadStackSize)

  • If the Java application is being started with a Windows service wrapper, test to see if the issue can be reproduced when the Java application is started outside the wrapper.

  • Solaris: Ask if the JVM is running on a Solaris zone.

  • Solaris: Even if the JVM is running in 32-bit mode, sun.arch.data.model: 32, Solaris can map 4GB per process, contrary to Windows and Linux.

  • Sun JDK 1.5 update 7 and 1.6: Add the following JVM option to capture pmap output when OutOfMemoryError happens to see address space information:

      -XX:OnOutOfMemoryError="pmap %p > pmap.out"

    Setting that in run.conf for JBoss can break parsing of the JAVA_OPTS being applied so in this case set it directly in run.sh instead:

      while true; do
     if [ "x$LAUNCH_JBOSS_IN_BACKGROUND" = "x" ]; then
        # Execute the JVM in the foreground
        "$JAVA" -XX:OnOutOfMemoryError="\"pmap %p > pmap.out\"" $JAVA_OPTS \
           -Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" \
           -classpath "$JBOSS_CLASSPATH" \
           org.jboss.Main "$@"
        JBOSS_STATUS=$?

  • On Windows, use Content from technet.microsoft.com is not included.Content from technet.microsoft.com is not included.http://technet.microsoft.com/en-us/sysinternals/dd535533.aspx to get the address space layout of a process.

  • Determine address space limits by testing different max heap sizes with java -version. The address space for the max heap size is reserved by the JVM on startup, but just the min heap size is allocated (RAM). For example:

$ java -Xms128M -Xmx2048M -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode)

$ java -Xms128M -Xmx3036M -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

Note: Apparently the HP JVM has a throttle and just sets Xmx to the max possible value if you set Xmx higher than is physically possible.
  • Windows: Check if physical memory is exhausted. There have been cases and Red Hat support has reproduced this error on 64-bit Windows when RAM is exhausted.

JBoss

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.