"java.lang.OutOfMemoryError: unable to create new native thread" in Openshift Enterprise

Solution Verified - Updated

Environment

  • OpenShift Enterprise
  • ActiveMQ

Issue

  • Below errors were seen in wrapper.log using activemq at Openshift Enterprise:
 Exception in thread "ActiveMQ Transport Server Thread Handler: stomp://0.0.0.0:61613" java.lang.OutOfMemoryError: unable to create new native thread

Resolution

Note: For OCP 4, refer to Unable to create more than 1024 Threads in OCP4.

Steps

  1. Increase the cgroup pids.max (not ulimit) on the number of threads. Content from www.kernel.org is not included.Reference

  2. In file /etc/activemq/wrapper.conf change the value of below parameter to false under Java Additional Parameters section. It will try to use the unused thread from the pool instead of creating/using a dedicated thread per destination. For more information see Content from activemq.apache.org is not included.link:

wrapper.java.additional.8=-Dorg.apache.activemq.UseDedicatedTaskRunner=true  
to 
wrapper.java.additional.8=-Dorg.apache.activemq.UseDedicatedTaskRunner=false
  • Restart the activemq service.

Root Cause

  • The stack dumps show that there are hundreds and hundreds of threads with names of this form. Which suggests there are many number of JMS queues:
Queue:mcollective.reply.apsrNNNNN
ActiveMQ.Advisory.Consumer.Queue.mcollective.reply.apsrpNNNN
  • Running oo-stats command at frequent interval from the broker hosts may cause the ActiveMQ JVM to run out of threads.

Diagnostic Steps

Check the value of pids.max:

$ cat /sys/fs/cgroup/pids/pids.max

Monitor the current number of pids:

$ cat /sys/fs/cgroup/pids/pids.current
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.