System hangs with a lot of threads blocked at java.lang.Class.forName0(Native Method) with no indicator of where the lock is held

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)
  • Oracle JDK 1.6
  • OpenJDK 1.6

Issue

There are a number of threads which appear to be blocked, but there is no indication of what is holding the lock looking at the thread dumps
All the affected threads have an identical stack trace 

	"ajp-0.0.0.0-8029-932" daemon prio=10 tid=0x43373800 nid=0x60dc waiting for monitor entry [0x34dfd000]
	   java.lang.Thread.State: BLOCKED (on object monitor)
		at java.lang.Class.forName0(Native Method)
		at java.lang.Class.forName(Class.java:186)
		at com.myclass.Validator$Criteria.isPermited(UserValidator.java:309)
		at com.myclass.Validator.isPermitted(UserValidator.java:340)
	        .. .. ..

Resolution

  • In the case of a ClassNotFoundException being the root cause, fix the underlying code.
  • If Class.forName() is being invoked in a loop in a business method, consider caching the data, if applicable, to prevent repetitive invocation at runtime.

Root Cause

Class initialiser call blocking, and stalling the Class.forName call

The threads are blocked for so long because the code is using the same loader and trying to load the same class, which is why the threads are being blocked until resolved one by one.

In the case of a ClassNotFoundException (CNFE) being the underlying issue leading up to the blocked threads, the CNFE is not being thrown quickly like any other runtime exception because for each forName call, it has to look through the current directory, which involves making an OS call and reading the directory node over and over, which means I/O starts to become a significant barrier.
If you make one small change, lilke bundling the class in question in a .jar, it will speed up significantly as it can now mmap the jar and search much faster. ( Of course most production apps already have the classes jared up.)

In Java 7, Classloader by default is registered as parallel capable.

However loading is still synchronized on a pair of classloader and class name: Content from docs.oracle.com is not included.Content from docs.oracle.com is not included.http://docs.oracle.com/javase/7/docs/technotes/guides/lang/cl-mt.html

Diagnostic Steps

  • Add additional logging to the method in question "(calling forName for class"+customClassName) . Basically something that will help pinpoint where the problem lies.
  • Add the (%t) parameter to the log4j conversion pattern, so it can be cross-referenced it with thread dumps, to determine the last call of the method before the thread becomes stuck.
SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.