JBoss hot deploying EJB3 deployments causing 'java.net.SocketException: Too many open files'

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)

  • EJB3 deployments

Issue

  • We deploy many times during the day using an automated build tool. At least 4 times a day per application with a total of 2 applications, so we're looking at minimum of 8 hot deployments a day.
  • Seeing the following error:
    java.net.SocketException: Too many open files

Resolution

The option -XX:SoftRefLRUPolicyMSPerMB=0 can be set when starting JBoss to force SoftReferences to be cleared. Further information is available at Content from java.sun.com is not included.Content from java.sun.com is not included.http://java.sun.com/docs/hotspot/HotSpotFAQ.html.

Root Cause

As part of the redeployment process the JVM supplied ObjectStreamClass is used to process the EJB3 proxies as they are bound to JNDI. Tthis class caches a reference to the class using a SoftReference:

Content from java.sun.com is not included.Content from java.sun.com is not included.http://java.sun.com/j2se/1.5.0/docs/api/java/io/ObjectStreamClass.html

Content from java.sun.com is not included.Content from java.sun.com is not included.http://java.sun.com/j2se/1.5.0/docs/api/java/lang/ref/SoftReference.html

This SoftReference causes the Classloader to be kept in memory until there is a demand for the memory; however, the Classloader also keeps files open which are not detected by garbage collection.

Diagnostic Steps

  • Identify the JBoss process id (e.g. using jps) and look for the process 'Main'.
  • Output the open files using 'lsof -p <jboss_pid>'
  • Check if there are a large number of open files under 'tmp/deploy'
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.